Ball hit detector using OpenCV

Recently, I happened to participate in one interesting project. My sister is studying for a designer at BHSD, and they were given the task of creating a project on Street Interactive. The idea was chosen quite simple. An animation of a moving bear is shown on the screen, everyone is invited to get into it from a slingshot with an impromptu snowball. The result is shown in the video, who are interested in the technical implementation, welcome to cat.

Description

The original idea was to use kinekt for tracking. It seemed that for such a task, the kinekt would be perfect - it has a good built-in camera, and also allows you to track the depth (and determine with some accuracy the position of the body in three-dimensional space). However, after a short test, I had to abandon the use of kinekt. It does not allow you to track objects at a long distance, and in addition, the bright light from the projector interferes with its sensors.

Then I got the idea to use a regular webcam for tracking. Position the camera near the projector and point it at the screen. Use it to track the position of the ball in the plane of the screen. But there was one more problem - to determine the moment of collision of the ball with the wall. As an option, an Arduino with a motion sensor was considered. However, in the end, it was decided to use a second camera located near the screen as a motion detector. With it, you can capture the moment when the ball flies to the screen, and take the coordinates of the impact a few milliseconds after this moment.

The software implementation was decided to be done in C ++ using the OpenCV library. It allows not to reinvent the wheel, but to use ready-made functionality for receiving images from the camera, and its subsequent processing.

Ball tracking

To determine the coordinates of the ball, I used the following algorithm.
1) Converted an image from RGB representation to HSV. This makes it easier to identify similar colors, because unlike RGB, HSV stores color tone, saturation and brightness in a separate channel.
2) Translated the image into binary (bitmap). Those colors that are closest to the required color (the color of the ball) are converted to white. The rest is in black.
3) Filtered the noise with a median filter.
4) Determined the average coordinate and the number of white pixels. If the quantity is greater than the threshold value, then there is a ball in the frame.
The resulting code is:

        clr=ballColor;
        frame=cvQueryFrame(capture); // Получаем изображение с камеры
        cvtColor(frame,frameHSV,CV_BGR2HSV); // Переводим в HSV
        inRange(frameHSV,Scalar(clr-ballThres,120,120),Scalar(clr+ballThres,255,255),frameBitmap);
                                                          // Переводим в bitmap
        medianBlur(frameBitmap,frameBitmap,5); // фильтруем шумы
        for(int i = screenLeft; i < screenRight; i++) {
            for(int j = screenUp; j < screenDown; j++) { 
                cl=frameBitmap.at(j,i);
                if (cl!=0) { // Находим центр тяжести
                    x+=i;
                    y+=j;
                    n++;
                }
            }
        }
        if (n>ballDifNum) { // Отсекаем некоторые случайные сробатывания
            x/=n;
            y/=n;
        }

Hit definition

To determine that a collision occurred, I used a second camera as a motion detector. To do this, I determined the difference between the previous frame and the current one, and if the difference is higher than the threshold value, then the ball flew into the frame:

        frame=cvQueryFrame(captureHit);
        cvtColor(frame, frame, CV_RGB2GRAY );   // из цветного в серое
        GaussianBlur(frame,frameCurrent,cv::Size( 3, 3 ), -1);  // убираем шумы
        absdiff(frameCurrent,framePriv,mask);  // смотрим разницу между кадрами
        framePriv=frameCurrent.clone(); // сохраняем предыдущий кадр
        threshold(mask,mask,motionTreshold,255,cv::THRESH_BINARY);
        if (countNonZero(mask)>motionDifNum)
              ttl3=detectDelay; // если мячик влетел в кадр, запускаем обратный отсчёт до столкновения

Result

As a result, having the coordinates of the ball and information about the collision, the program transferred them to the second program, which is responsible for the animation. The result of the project is presented in the first video. And below you can look at technology testing at home.

References

balltracking.cpp - Complete listing of
wikipedia.org - About OpenCV on Wikipedia
robocraft.ru - OpenCV step by step. OpenCV lessons in Russian from Cheshire Cat.
aishack.in - Tracking colored objects in OpenCV
britishdesign.ru - British Higher School of Design