Hello!

Probably, it is not a secret to anyone that lately cloud services of video surveillance are gaining popularity. And it's clear why this is happening, the video is “heavy” content, which requires infrastructure and large amounts of disk storage to store. The use of a local video surveillance system requires funds for operation and support, as in the case of an organization that uses hundreds of surveillance cameras, and in the case of an individual user with several cameras.

Cloud surveillance systems solve this problem by providing customers with an already existing video storage and processing infrastructure. For a cloud surveillance client, simply connect the camera to the Internet and link it to your account in the cloud.

There are several technological ways to connect cameras to the cloud. Undoubtedly, the most convenient and cheap way - the camera is directly connected and works with the cloud, without the participation of additional equipment such as a server or a registrar.

For this, it is necessary that the software module working with the cloud be installed on the camera. However, if we talk about cheap cameras, then they have very limited hardware resources, which are almost 100% taken by the native firmware of the camera vendor, and the resources required for the cloud plug-in are not. Developers from ivideon have devoted an article to this problem , which states why they cannot install a plug-in on cheap cameras. As a result, the minimum price of the camera is 5000r ($ 80 dollars) and millions of money spent on equipment.

We have successfully solved this problem. If it is interesting how - well under kat

A bit of history

In 2016, we started the development of a cloud video surveillance platform for Rostelecom.

In the software part of the cameras, at the first stage, we went “standard” for such tasks: we developed our own plugin, which is installed into the vendor's firmware firmware and works with our cloud. However, it is worth noting that during the design we used the most lightweight and efficient solutions (for example, plain C implementation protobuf, libev, mbedtls and completely abandoned convenient but heavy boost-type libraries)

Now, there are no universal integration solutions on the IP camera market: each vendor has its own way of installing the plug-in, its own set of APIs for running the firmware, and a unique update mechanism.

This means that for each camera vendor it is necessary to individually develop the bulk layer of integration software. And at the time of the start of development, it is advisable to work only with the 1st vendor to focus the team’s efforts on developing the logic of working with the cloud.

The first vendor was chosen Hikvision - one of the world leaders in the camera market, providing a well-documented API and competent engineering and technical support.

On Hikvision cameras, we launched our first pilot project, cloud video surveillance, Video Comfort.

Almost immediately after launch, our users began to ask questions about the possibility of connecting to the service of cheaper cameras from other manufacturers.

I dropped the variant with the implementation of the integration layer for each vendor almost immediately - as a poorly scalable and having serious technical requirements for the camera hardware. The cost of the camera that meets these requirements at the entrance: ~ $ 60-70

Therefore, I decided to dig deeper - to make my own firmware for the cameras of any vendors. This approach significantly reduces the requirements for camera hardware resources - because the cloud layer is an order of magnitude more efficiently integrated with the video application, and there is no excess unused fat in the firmware.

And what is important, when working with a camera at a low level, it is possible to use hardware AES, which encrypts data without creating additional load on a low-power CPU.

At that moment we had nothing at all. Nothing at all.

Almost all vendors were not ready to work with us at such a low level. There is no information about circuitry and components, official SDK chipsets and sensor documentation are not.
There is also no technical support.

Answers to all questions had to be obtained by reverse engineering - by trial and error. But we managed.

The first camera models on which we stuffed bumps were the Xiaomi Yi Ants, Hikvision, Dahua, Spezvision, D-Link cameras and a few super cheap unnamed Chinese cameras.

Equipment

Chambers on the Hisilicon 3518E chipset. The hardware characteristics of the cameras are as follows:

We started with them.

Now we support Hisilicon 3516/3518 chipsets, as well as Ambarella S2L / S2LM. The number of camera models is dozens.

Firmware composition

uboot

uboot is the bootloader, after power up is loaded first, initializes the hardware and loads the linux kernel.

The boot script of the camera is quite trivial:

bootargs=mem=38M console=ttyAMA0,115200 rootfstype=ramfs mtdparts=hi_sfc:256K(boot),64K(tech),4096K(kernel),8192K(app),-(config) hw_type=101
bootcmd=sf probe 0; sf read0x820000000x500000x400000; bootm 0x82000000; setenv bootargs $(bootargs) bkp=1; sf read0x820000000x4500000x400000; bootm 0x82000000

Of the features, it is called twice bootm, more on this later, when we get to the update subsystem.

Pay attention to the line mem=38M. Yes, yes, this is not a typo - the Linux kernel and all-all applications have only 38 megabytes of RAM available.

Also next to the uboot is a special block, called reg_info, in which the low-level DDR initialization script and a number of SoC system registers are located. The content reg_infodepends on the camera model, and if it is not correct, the camera will not even be able to load the uboot, but will freeze at the very early stage of the download.

The first time, when we worked without vendor support, we simply copied this block from the original camera firmware.

Linux kernel and rootfs

The cameras use the Linux kernel, which is part of the chip's SDK, usually these are not the freshest 3.x kernels, so we often have to deal with the fact that the additional hardware drivers are not compatible with the used kernel, and we have to back them up to the kernel cameras.

Another problem is the size of the kernel. When the size of FLASH is only 8MB, then each byte per account and our task is to carefully disable all unused functions of the kernel in order to reduce the size to a minimum.

Rootfs is a basic file system. It includes busyboxthe wifi module driver, a set of standard system libraries, the type libldand libc, as well as our development software, which is responsible for the logic of controlling LEDs, managing network connections, and updating firmware.

The root filesystem is connected to the kernel as initramfs and as a result of the build we get one file uImage, in which there is both a kernel and rootfs.

Video application

The most complex and resource-intensive part of the firmware is an application that provides video-audio capture, video coding, adjusts image parameters, implements video analytics, such as motion or sound detectors, controls PTZ and is responsible for switching day and night modes.

Important, I would even say a key feature - how the video application interacts with the cloud plugin.

In traditional solutions, “vendor firmware + cloud plugin”, which cannot work on cheap hardware, video inside the camera is transmitted via the RTSP protocol - and this is a huge overhead: copying and transferring data through a socket, extra syscalls.

In this place we use the mechanism of shared memory - the video is not copied and is not sent through the socket between the components of the camera software, thereby optimally and carefully using the modest hardware capabilities of the camera.

Update subsystem

The subject of separate pride is the fault-tolerant subsystem of the online firmware update.

I will explain the problematic. A firmware update is technically not an atomic operation and if a power failure occurs in the middle of the update, then the flash memory will contain a part of the "underwritten" new firmware. If you do not take special measures, the camera after that will become a "brick" that you need to carry to a service center.

We coped with this problem. Even if the camera is turned off at the time of the update, it automatically and without user intervention will download the firmware from the cloud and restore work.

We analyze the technique more:

The most vulnerable moment is overwriting the partition with the Linux kernel and root filesystem. If one of these components turns out to be damaged, then the camera will not boot at all beyond the uboot bootloader, which cannot download firmware from the cloud.

So, we need to ensure that the camera has a working kernel and rootfs at any time during the update process. It would seem that the simplest solution would be to permanently store on the flash memory two copies of the kernel with rootfs and, in case of damage to the main kernel, load it from a backup copy.

A valid solution - however, the kernel with rootfs takes about 3.5MB and for a permanent backup you need to allocate 3.5MB. On the cheapest cameras just do not have as much free space under the backup kernel.

Therefore, to backup the kernel during the firmware update, use the application partition.
And to select the desired partition with the kernel, two commands are used bootmin uboot - in the beginning we try to load the main core and if it is damaged, then the backup one.

This ensures that at any time the camera will have the correct kernel with rootfs, and it will be able to boot and restore the firmware.

CI / CD build system and firmware deployment

To build the firmware, we use gitlab CI, in which firmware for all supported camera models is automatically collected, after the firmware is assembled, it is automatically applied to the camera software update service.

From the firmware update service, the firmware is delivered to our QA test chambers, and upon completion of all testing phases, to the users' cameras.

Information Security

It is not a secret for anyone that in our time information security is the most important aspect of any IoT device, including cameras. On the Internet, Mirai-type botnets walk around, hitting millions of cameras with standard firmware from vendors. With all due respect to the camera vendors, I can not help but note that the standard firmware has a lot of functionality that is not required to work with the cloud, but contains many vulnerabilities that botnets use.

Therefore, all unused functionality in our firmware is disabled, all tcp / udp ports are closed, and when updating the firmware, the software digital signature is checked.

And besides this, the firmware is regularly tested in the information security laboratory.

Conclusion

Now our firmware is actively used in video surveillance projects. Perhaps the most ambitious of them is the broadcast of the vote on the day of the election of the President of the Russian Federation.
The project involved more than 70 thousand cameras with our firmware, which were installed at polling stations in our country.

Having solved a number of difficult, and in some places, even at that time practically impossible tasks, we, of course, were greatly pleased as engineers, but beyond that, and saved millions of dollars in purchasing cameras. And in this case, the savings are not only words and theoretical calculations, but the results of a tender for the purchase of equipment. Accordingly, if we talk about cloud-based video surveillance: there are two approaches - to strategically base on low-level expertise and development, having a huge savings on equipment or use expensive equipment, which, if you look at consumer characteristics, is practically no different from similar cheap ones.

Why is it strategically important to decide on the choice of approach to the integration method as early as possible? When developing a plug-in, developers are laid on certain technologies (libraries, protocols, standards). And if a set of technologies is chosen only for expensive equipment, then in the future an attempt to switch to cheap cameras is likely to take at least an insanely long time or fail altogether and return to expensive equipment.