How we built the first prototype

Hi Habr! This is our second post, it opens a series of technical publications. We decided to tell in more detail about the CamIRa architecture , the components and technologies used, and also about how we came to this. It was time to share something with the community, especially since a lot of things had already accumulated that we had to deal with during the implementation of our first prototype. In this article we will tell you how you have gone from a simple program with one IR image to a full-fledged cloud system with a personal account.
Interested, we ask for a cat!

Prototyping

From the appearance of an idea and before it becomes something tangible, a lot of time can pass, it can take months, sometimes years! In order to reduce these very years to at least a month or two, and there is prototyping.
Today there are many things available for rapid prototyping, many of which are available to almost everyone. These are all kinds of single-board computers and amateur boards with programmable microcontrollers: Raspberry Pi, Arduino, Intel Galileo and others. For fast manufacturing of the case 3D printing is used.
From the very beginning, we owned these tools, so we only had to find the right IR sensor and put everything together.

Sensor purchase

The suitable sensor option for us was the IR-Blue RHWorkshop Melexis MLX90620 used in the device . This is a rather primitive sensor with a 16x4 matrix, but already connected to the board with Bluetooth, and all source codes for it are available on GitHub . For prototyping, that's it. We decided to order a ready-made device, especially since it was much more difficult to order only a sensor from the manufacturer.
Many of Habr’s readers know how difficult it is to find something rare in our stores, you have to order because of the “hillock”, and also wait weeks or even months (hi to Russian Post). We were ready for this and much more from our native post, but we could not even think that something could happen at the delivery service at the London airport!
By that time, the company had kicked off long ago, and the remnants of IR-Blue were sold on the website ( RHWorkshop ). We ordered one assembly and one as a set of components for self-assembly, it is a bit cheaper, and we wanted to use the sensor from it, we would not even have to solder it.
According to the seller, the transport used by him delivers to Moscow in 14 days. The seller sent us a tracker, on it we saw that the package arrived in London just a couple of days after it was sent from the USA! Great, we thought: what speed. After that, we stopped monitoring the tracker, relaxing and deciding that everything was in order. 14 days have passed, there is no parcel, we track the tracker, it shows that the parcel is still in London! They wrote to the seller, since he made out the dispatch, and only he can kick the transport company. After a brief correspondence with him, it turned out that customs officers in London did not agree with the declared price for this type of parcel. And most importantly, they don’t do anything, they would send it back and that’s all, but they just don’t pass it on, keep it at home! Having discussed the situation with the seller, we decided to withdraw, repack and send in two parcels.
Two weeks later, we received it. The device worked stably, but did not differ in the quality of the case and its materials. However, like most products with kickstarter. But the software for him was insanely buggy and capricious on different phones and different versions of Bluetooth. After long dances with a tambourine, we managed to get a picture through native software.

Assembly

After playing around with the device, we threw a list of components for the first CamIRa prototype : -
Raspberry Pi model B -
Arduino Uno -
Servo drives FS90 2 pcs. fs90 -Service
bracket for servos 9g servo bracket
-RaspiCam raspicam
-Sensor 16x4 Melexis MLX90620 as part of the IR-Blue device

The device’s brain is the Raspberry Pi. General management of the device, receiving commands from the server, receiving video from RaspiCam, interacting with Arduino is a Java application that runs on Malinka. Arduino is responsible for controlling the servomotors, as well as reading data from the IR sensor (in the first versions, the data from the sensor was obtained directly via Bluetooth), which is controlled by Raspberry via UART / RS-232. For these purposes, I even had to develop my own text-based interaction protocol Raspberry-Arduino based on RS-232.
In the assembly, everything looks like this:



On top of the disassembled IR-Blue device, RaspiCam is attached with an adapter for a wide-angle lens, printed on a 3D printer. All this is mounted on a bracket with servomotors and connected to the Arduino and Raspberry Pi. By the way, we had to make our bike with a lens, since the camera’s native viewing angle (FOV) was 50 degrees, which is even less than the IR sensor. There is an upgrade instruction on the bourgeois website: wiki.raspberrytorte.com/index.php?title=Camera_Module_Lens_Modifcation . A small catch was that the lens mounts have to wait a long time with e-bay, and in addition, the distance between the mounting holes of the holders does not correspond to the distance on the RaspiCam board. Therefore, a 3D model was quickly drawn and printed.



Anyone wishing to increase the viewing angles of RaspiCam, here is the model adapter: file . You need to print it on a printer, cut the M12x0.5 thread, carefully unscrew the native lens from the camera with tweezers, install over the adapter and screw such lenses tyts or tyts2 there . Our experiments have shown that the Daiwon BL2120 [DW2162] lens gives the greatest angle and the best quality. It turned out FOV = 110 degrees, which is quite enough for an extensive coverage of a 4x5 m2 room, plus RaspiCam gives a Full HD picture.

Software component

Having finished with iron, they began to write software for him. Initially, the sources of the IR-blu program for an iPhone from a github were taken for study. But, as often happens, understanding someone else’s code is more difficult than writing your own, so the IR-blue code was not used. Armed with a datasheet for the sensor, we started developing (by the way, a little later for the processing of the IR image it was tempting to take the Habr Post project code , mentioned in our first post, but in order not to waste time, the sources were practically not even inspected). Our first program for this “monster” looked like this (these are three fingers):



Subsequently, it formed the basis of our utility program, in which the calibration of infrared sensors and servomotors is performed. After several weeks of operation, the program window began to look like this:



In the screenshot of the program, the image of the mug with boiling water: top left of the sensor in the center position, in the center - a composite image for several movements of the sensor (IR image).
Gradually debugging and testing functions directly from a PC, we began to transfer functionality to the server. Now Raspberry works with the server via HTTP / S (for streaming video) and WebSockets / wss (for sending commands). The server part consists of several services - a video server and a WebSocket server (both own development in java, spin on Tomcat) are deployed on Azure (thanks to BizSpark), a web server with a personal account is also a Java application using the Play framework! , a set of Bootstrap + jQuery and native JavaScript, deployed on GAE.
As a result, the general interaction architecture looks like this:



Not a screenshot below shows the result of the device with a 4x4 Omron D6T-44L-06 ( Omron ) sensor . In a browser in your account, a thermal image is superimposed on top of the streaming video (online).



In the next article we will talk more about the software part, in particular about the implementation of streaming video using RaspiCam, Raspberry Pi and various libraries for converting and processing streaming video. The next post will be useful to everyone who would like to make a full-fledged video surveillance system from Raspberry. If someone is not familiar with us, we recommend reading our first post