Hold on tightly to the steering wheel ... Our project to monitor the condition of drivers

Every motorist knows that drivers tend to get tired. It happens that your eyes stick together even when you are driving home from work, and truckers have one of the main professional problems: thousands of accidents occur every year due to people falling asleep at the wheel. The thing is that truckers need to constantly maintain a balance between rest and safe driving on the one hand, and fatigue with higher earnings on the other. Great temptation to sleep less, but faster to deliver the goods and get a new order. Not to mention the fact that drivers have their own personal lives and they do not always go to sleep on a flight.

To make the roads safer, we launched our project to monitor the condition of drivers: according to customers' requirements, we create software and hardware solutions that can track the driver’s health on the road and prevent the stage of falling asleep so that they don’t “wake up” in the next world.

Vigilant Eye

The main idea is to use the camera to track the fatigue and dangerous reactions of drivers. The device first collects statistics on how a particular driver behaves while driving. And if, on the basis of these statistics, the system decides that the driver is showing signs of fatigue or is about to see dreams, she will signal this to him. If the driver often began to nod or reacts to events like a sleepy fly, the system also informs the dispatcher about this.

Using video analytics, we monitor how focused the driver is and whether he is looking at the road. The camera constantly captures the face and hands, and the system tracks a number of key points. We expect that the driver’s eyes are always open, his eyes are directed towards the windshield or side mirrors. If the eyes close, or the head leans forward and the chin falls on the chest, or the face turns away from the windshield for a long time, then the system gives a signal.

Of course, this tracking and signaling does not work all the time, but only when the car is moving, a GPS sensor is used for this. We also provided for the possibility of forcibly turning the system on and off, because the situations are different. But at the same time, we compare the duration of the machine and the time when the system is turned off. If necessary, the device can inform the dispatcher that the driver has been traveling with the “control” off for a long time.

Implementation

The project consists of two parts: hardware and software.

How does the system determine that the driver is falling asleep now? Using the OpenCV library, which is based on machine learning. OpenCV allows you to define key points in the video stream: chin, hands, eyes (including their size and breadth of disclosure). And based on these parameters, the system determines whether the driver is tired or not.

The hardware consists of the Waveshare RPi Camera (F) and the Raspberry Pi, assembled in a single housing. Creating hardware is more difficult than it might seem at first glance: you do not need to connect and solder the components, but think through their combination so that the device works reliably for a long time and without failures. In fact, this is a separate integration task. And integration risks are usually the highest. After all, it is one thing when the software package runs on your laptop, under your OS and with your limitations, and another thing when it is transferred to the Raspberry Pi, under another OS, where the processor is much weaker and less memory.

Of course, if the product is put in series, then it will no longer be implemented on the Raspberry Pi. The piece of iron is weak, its power is not enough to process the video stream in full resolution and to recognize the face, so with such hardware you have to go to tricks.

In addition, we implemented a night mode, so that the system could wake up the driver in the dark, when people often fall asleep. That is, I had to teach the computer how to handle video from an infrared camera. Here we were not pioneers, a similar problem is also solved in smart home systems. To determine the driver’s fatigue, you don’t need to know details about his appearance: hair color, eyes, skin tone. We only need to clearly determine the position of the hands and key points on the face. Oddly enough, it turned out to be even easier to work with eyes in infrared mode than in normal. If you've ever seen clips shot with automatic cameras with animals in the wild, you know that with a good angle, their eyes glow like laser ones.


This effect turned out to be very useful to us: it is easier to determine the fact of closed eyes. Of course, an additional adjustment of all algorithms was required, because the spectrum of the image is completely different, but, in general, the eyes are more confidently recognized.

But in infrared mode, the system recognizes worse situations when the driver speaks on the phone, pressing his ear to his shoulder. This is also a dangerous situation, traffic rules are explicitly forbidden to talk on the phone while driving.

Of course, like any machine learning model, ours also does not work with absolute accuracy. But you need to understand well what exactly recognition errors depend on. For example, we have not tested our system on drivers with a non-European appearance. Perhaps in this case, the proportion of accurate eye recognition and fatigue will decrease. However, during internal testing, the system worked exactly in 8-9 cases out of 10.

* * *

Today, there are a lot of good algorithms in the public domain, and we took ready-made ones, which allowed us not only to solve the problem, but also to save a lot of time. Now you can quickly find a model suitable for solving a unique task, and quickly refine it. Of course, this requires skills and experience in solving computer vision problems and quick adaptation. Then it takes only one or two days to get a prototype on a laptop. And further work depends on the task, on available capacities, storage and other factors. For example, you can quickly refine our system so that it plays a different role: let's say, as a security device, which ensures that only specific people enter the room at a certain time of the day. That is, you can quickly create a solution for a variety of tasks related to computer vision.