installing opencv

OK. We set up our raspberry pi. We built a car around it. We made a serviceable GUI for it. We toyed with PWM. Now what? Well, we spent some money on a good usb camera, so how about image processing? Again, something relatively new for me. There is a fantastic collection of code called opencv, so let's use that. "OpenCV (Open Source Computer Vision) is a library of programming functions mainly aimed at real-time computer vision." To install it on the Raspberry Pi is a little bit of work, but with some googling, and following directions, it worked. But given the pi's speed and limited RAM, anything intensive is probably best done on another box. Here are the install instructions that worked for me:

sudo apt-get install build-essential
sudo apt-get install cmake git libgtk2.0-dev pkg-config libavcodec-dev libavformat-dev libswscale-dev
sudo apt-get install python-dev python-numpy libtbb2 libtbb-dev libjpeg-dev libpng-dev libtiff-dev libjasper-dev libdc1394-22-dev

mkdir opencv
cd opencv
wget -O opencv.zip https://github.com/opencv/opencv/archive/3.4.1.zip
unzip opencv.zip

wget -O opencv_contrib.zip https://github.com/Itseez/opencv_contrib/archive/3.4.1.zip
unzip opencv_contrib.zip


cd opencv-3.4.1/
mkdir build
cd build

cmake -DCMAKE_BUILD_TYPE=RELEASE \
    -DCMAKE_INSTALL_PREFIX=/usr/local \
    -DINSTALL_C_EXAMPLES=ON \
    -DINSTALL_PYTHON_EXAMPLES=ON \
    -DOPENCV_EXTRA_MODULES_PATH=~/opencv/opencv_contrib-3.4.1/modules \
    -DBUILD_EXAMPLES=ON \
    -DBUILD_DOCS=ON  ..

make
sudo make install
sudo ldconfig

# now install matplotlib:
sudo apt-get install python3-matplotlib

Be warned that 'make' took 7 hours on my raspberry pi. So it was important to power the pi from the micro-USB cable rather than the battery pack on the car. For a speed up, it is possible to run jobs in parallel, eg:

make -j4

but on the pi you run the risk of running out of RAM. So I opted for the slower version.

Now on to some elementary image processing. First up, average images from our camera. The output from the camera can be quite good, but is noisy in lower light levels. Average tidies this up a bit:

import numpy as np
import cv2

count = 20

cap = cv2.VideoCapture(0)

# create a list of first count frames:
img = [cap.read()[1] for i in range(count)]

# rotate images:
rot_img = [cv2.flip(i, -1) for i in img]

# convert all to float64:
float_img = [np.float64(i) for i in rot_img]

# find the average:
mean_img = float_img[0]
for k in range(1, count):
    mean_img += float_img[k]
mean_img /= count

# Convert back to uint8:
int_mean_img = np.uint8(np.clip(mean_img, 0, 255))

# show output:
cv2.imshow('initial', rot_img[0])
cv2.imshow('average', int_mean_img)

# save output:
cv2.imwrite('raw-image.png', rot_img[0])
cv2.imwrite('ave-image.png', int_mean_img)

# tidy up and quit:
cv2.waitKey(0)
cv2.destroyAllWindows()

Next up, unscaled and scaled image similarity. These two functions return 0 for completely distinct, 1 for identical, and values in between otherwise. First, unscaled:

import numpy as np
import cv2

def image_simm_unscaled(im1, im2):
    # convert all to float64:
    fim1 = np.float64(im1)
    fim2 = np.float64(im2)

    wf = np.sum(np.absolute(fim1))
    wg = np.sum(np.absolute(fim2))
    wfg = np.sum(np.absolute(fim1 - fim2))

    if wf == 0 and wg == 0:
        return 0
    return (wf + wg - wfg) / (2 * max(wf, wg))

Now scaled:

import numpy as np
import cv2

def image_simm_scaled(im1, im2):
    # convert all to float64:
    fim1 = np.float64(im1)
    fim2 = np.float64(im2)

    s1 = np.sum(fim1)
    s2 = np.sum(fim2)

    if s1 == 0 or s2 == 0:
        return 0

    wfg = np.sum(np.absolute(fim1 / s1 - fim2 / s2))

    return (2 - wfg) / 2

Next, combine average and image-similarity into one function, simm-average. The idea being, we only add it to our average-image if it passes a similarity threshold. This helps with the ghosting problem I had with a simple average, but seems not to 100% solve that issue. Maybe with tweaks to the value of IMAGE_SIMILARITY we could completely remove ghosting?

IMAGE_SIMILARITY = 0.75

def create_simm_average_camera_image(count):
    # create a list of first count frames:
    img = [cap.read()[1] for i in range(count)]

    # rotate images:
    rot_img = [cv2.flip(i, -1) for i in img]

    # convert all to float64:
    float_img = [np.float64(i) for i in rot_img]

    # find average of similar images:
    mean_img = float_img[-1]
    i = 1
    for k in range(1, count):
        tmp_img = float_img[k]
        r = image_simm_scaled(mean_img, tmp_img)
        if r > IMAGE_SIMILARITY:
            mean_img += tmp_img
            i += 1
    mean_img /= i
    print('we averaged %s images' % i)

    # Convert back to uint8:
    int_mean_img = np.uint8(np.clip(mean_img, 0, 255))

    return int_mean_img

In the next post, something more interesting, edge-enhance.