“Smart” video surveillance: what will life be like under cameras with artificial intelligence

The cameras are watching us almost continuously, but there is little sense in this. If a person does not analyze the picture, the camera remains just a device producing terabytes of hours of unsuitable stream. An alternative is to equip the camera with AI tools. And such a video surveillance system will be able to replace the security guard sleeping in front of the monitor, the boss in the office and the marketer in the supermarket. We tell you exactly how.

A “smart” camera is a conditional concept, and the mind itself in most cases is not mounted on the camera itself, but is installed on a server where the video stream from the camera is analyzed using artificial intelligence technology. The camera does not have the computing power to do complex image analysis. Hereinafter, we will discuss precisely such gadgets, therefore, a “smart” camera is just the “eye” of a truly smart computer, which has all the intellectual load.

Room Scout: What is the Big Brother useful at home

In 2014, Google announced the Nest Cam, a small camera designed to monitor home security. She was able to recognize people's faces, see in the dark, hear voices, and also transmit her impressions of what was happening in the hosts' mobile application. With the advent of the voice-activated speaker Home from Google in 2016, it became possible to integrate a smart camera into the ecosystem of other smart gadgets from Good Corporation.

And at the turn of 2017 and 2018. Small startups entered the market, say Lighthouse, created specifically for the development and promotion of “smart” cameras, which now have several dozen on the market. How do these devices work?


Inside, the Nest Indoor IQ Cam has a six-core processor, 4K sensor, microphone and speaker. Source: Nest.com

The “smart” camera creates a dynamic 3D model of the room and all objects in it. Any change in the situation (movement, movement, appearance of a new object) is recorded and reflected in the model. Machine learning allows the camera to classify both objects and their actions. The longer the camera works, the better it understands who is in front of it, recognizes its own and others, learns to distinguish between a child and, say, a dog. Special attention is paid to people - the camera recognizes faces, creates a base of visitors with their photos in high resolution and at different angles, compares them and numbers them. Sometimes the camera asks the owner through the application to clarify the status of a person, showing photos of her embarrassing faces. The camera analyzes what is happening non-stop, transfers data to the cloud service and to the client application.

Such cameras both as part of a smart home and outside it make it possible to implement a number of new features and scenarios. If your beloved dog woke up earlier than you and started wandering around the house, the “smart” camera will not wake you with good morning wishes, because it distinguishes you from the animal. If the wind rocked the tree outside the window, the camera does not call the police, believing that the house is breaking, because cars or natural objects are also identified by it. A conventional motion sensor could easily respond to such events, causing unnecessary trouble.

If you are not at home, and, for example, a courier should come to you, you can remotely open the door for him, track his movement with Nest, tell through the speaker where to leave the parcel, and then politely say goodbye. Source: Google Nest YouTube channel.

But the smart neighbor pays attention to a new neighbor who came to borrow salt. When she sees a person for the first time, she remembers his face, and also sends a notification about the stranger to his home. Of course, if a neighbor in your absence borrowed salt or more expensive, the camera would give an alarm. If subsequently a neighbor swears by her mother that she was not going to steal anything, the camera will still remember her, as well as all the other guests who have distinguished themselves by unusual behavior - especially advanced gadgets distinguish between normal and emergency situations in the home.

Abnormal situation in the home. Source: Wyze YouTube Channel

When you returned home from work, you wanted to play the Xbox, but you find that the set-top box is not in place. Neighbor again? Courier? “No,” the camera will report. “This is your beloved mother, during her visit yesterday, put the gadget in the closet.” The camera can report the disappearance of the subject from the usual environment. When leaving home, you ask the camera to notify you of your mother’s visit in your absence. As soon as this happens, you will receive a message and call mom asking you not to touch the Xbox anymore.

The functionality described above is already the real capabilities of smart cameras, such as Nest, Wyze Cam, Arlo, Simplisafe, etc. In 2018, the market for such devices reached $ 7 billion, according to Strategy Analytics. By 2023, this figure may grow to $ 9.7 billion, and in "pieces" their sales may increase from the current 57 million to 120 million units over the same period.

Everything will be fine, wherever you go: smart cameras in the streets

Back in the 1980s, a camera distinguishing license plates for the first time helped to find a stolen car, but only in the 2010s “smart” cameras learned to analyze not one, but thousands of characteristics of the observed objects. So, in August 2017, the Chinese search engine Baidu announced that it was able to recognize various actions of people - from walking with a dog and washing windows to cutting down trees, etc. - for 300 thousand videos with an accuracy of 88%.

However, home smart cameras can do this, but they observe a stationary situation, and people are constantly moving on the street, and they shoot their different lenses. How to put together a single picture for analysis?

Toshiba has developed SATLYS technology. It identifies a specific person who has fallen under the lenses of different cameras. For this, artificial intelligence identifies a small number of distinctive features of the individual and compares them with the signs of other passers-by, who could get into the stream of neighboring cameras.


SATLYS makes street cameras work as a team so that you can build the path of any person using different broadcasts. Source: Toshiba

The computer does not take into account all the characteristic features, so as not to overload itself with comparisons of millions of features of appearance, but selects one or more significant ones. It is them he is looking for on other records. Moreover, the system is able to search for a person by an external attribute that can be entered into the system, as in the Google line, say, “red backpack”, “white dress”, “girl”, etc.


The main difficulty is that different cameras give a different image of a person due to a mismatch in the quality of the shooting, viewing angle, light location, etc. Toshiba has created a technology that can highlight a specific feature of an individual and look for a match with it in other video broadcasts. Source: Toshiba

So, street “smart” cameras can recognize us, understand what we are doing and where we are going. What does it give? For example, here’s what: leaving your house in the morning to get to your (“smart”, of course, what else!) Office, you didn’t attach importance to the stranger who revolved around the neighboring Bentley, but the “smart” camera on the house (or rather , the video surveillance system of which it is a part) has already transmitted information about it to the police, because it knows how to identify idlers. You smoke a cigarette and throw it on the sidewalk - you will have to pay a fine for this sinner, because systems can distinguish between offenses, even small ones.

You decide to go to the office by city electric train. At the train station, you are waiting for a train and you see a person staggering along the tracks. Perhaps he is drunk or he became ill - this is also reported to the station personnel by a video surveillance system that determines inappropriate human behavior in places of increased danger.

In the train, you talked nicely to a beautiful stranger, and when you parted, you found that your watch evaporated with it. The only thing you remember is the red backpack over the shoulders of the girl. A policeman at the station makes a request to the video surveillance system “red backpack”, “girl” and finds all the beautiful strangers with red backpacks that appeared today on the recordings. You point your finger at yours — the police will look for her.

Hitachi has created a video surveillance system for large public spaces - shopping centers or stadiums - which is able to track the movement of several people at once and analyze their appearance (hair length, color of clothes, etc.). Source: CGTN YouTube Channel

From the station to your favorite work some two kilometers, and you take a bike. You are driving yourself and you still don’t know that a Gelendvagen is rushing across under the control of a very busy bearded guy who ignores traffic signals. Smart cameras at the intersection noticed you and him a long time ago and sent you and other passers-by (and the police) a warning about a dangerous driver in a mobile application. Traffic lights hold off a red signal for pedestrians, and everyone safely avoids trouble, which cannot be said about the driver of a Gelendvagen. And here is my favorite office. But first, about the pictures of the beautiful cities of the future and reality.

In the harsh reality, the greatest progress in public systems of smart video surveillance has been achieved by the Chinese, who are implementing three projects in this area at once. Networks of cameras under the ominous name "Vigilant Eye" (more than 180 million devices), as well as Sky Net (no, this is not a joke; more than 20 million lenses) and "Safe Cities" (no, this is not irony; more than 2 million cameras) are not they only regularly monitor the inhabitants of the PRC, but they are already able to recognize and search for criminals. Moreover, the Chinese Big Brother is even able to punish citizens automatically. The transition to a red light entails photographing the intruder, recognizing his face and automatically publishing shame on the electronic board in the city.

But the integration of video surveillance in the urban Internet of things is still developing - it is more expensive and more complicated. So, in Detroit, one of the streets is equipped with a smart surveillance system associated with traffic lights and a special mobile application for citizens. It identifies careless pedestrians and reports them in notifications to drivers. It is also able to extend the green traffic light for cyclists rapidly approaching the transition.

One-eyed boss: the pros and cons of smart cameras at work

Cameras in the workplace have long been part of the office space, but in many ways they remain decorative if the boss is not sitting on the other side of the 24/7 lens. However, now it can be replaced by artificial intelligence.

In 2017, Microsoft introduced a comprehensive system for smart surveillance of the workspace. Cameras, computers and peripherals are connected to a remote “intelligent cloud” that uses AI to analyze what is happening in terms of safety and productivity. In this case, smart cameras work as part of the Internet of things ecosystem. In addition to analyzing more than 27 million different events in the picture, AI tools in the cloud receive signals from work computers, machine tools and other equipment.

How does it feel to work in such an office? The scenario may be like this. You go to work without any passes because the corporate camera knows you by sight. A colleague comes forward: he never greets, puts his mug on your documents and jokes about your mother. The smart camera considers the sour expression of your face and will transmit information about a possible conflict to the HR department. Immersed in thoughts of a new neighbor, you completely forget to attach a badge with a name and photo to a prominent place, as required by corporate rules. Go carelessly to the workplace, not knowing that you have already run into a fine, because the smart camera recorded your violation and transmitted information about it to the head of the department, who, after a few minutes, will remind you of the importance of complying with company rules (in a sharp and not very pleasant manner) because he received a notification on the smartphone about the employee without a badge. But cameras also follow him and know: lately he has been sleeping too often in front of the monitor off, and he forgets where he left the badge, although the “smart” camera then helps him find it.

A smart boss can be useful not only in the office, but also in the hospital: a patient with a heart condition decided to walk around the department. A smart camera captures this. A device that measures its heart rate also transmits information to the “smart cloud”. As soon as the patient’s heart begins to tire, an alarm comes to the post, and one of the nurses approaches the patient. Source: Microsoft YouTube channel.

Leaving work, you get a report on your performance during the day: based on the video data, the AI ​​compiled activity graphs, as well as a list of errors made.

We are far from a comprehensive workflow analysis system in which cameras play a leading role. While they mainly monitor compliance with safety regulations. For example, in the Australian branch of the international construction company Laing O'Rourke, a system for monitoring and warning of dangerous situations at a construction site has been introduced. As soon as smart cameras see a possible danger for workers, they send threat messages to their smartphones or smart watches.

But the analysis of employee productivity is mainly based not on direct observation of them, but on monitoring specific work operations. Such data is easier to “take” from computers and other working equipment than to observe an employee whose entire arm can only move the mouse.

Okay, it's time to move home from the office, but first you need to look into the store.

Under the gaze of marketers: why “smart” cameras in a shopping center

Cameras have long been watching thieves not stealing anything from the store. But for this, the "natural" intelligence of a security guard sitting all shift in front of the monitor is usually enough. Meanwhile, most often cameras see law-abiding customers, and this video information can be much more useful for retailers.

In particular, the American Walmart network realized this, which opened in April 2019 a store equipped with cameras, the image from which is processed by artificial intelligence. Cameras and sensors in the supermarket generate 1.6 TB of information per second, which is processed by the server directly in the store.


To impress visitors to a smart hypermarket, Walmart placed a data center that receives data from cameras and sensors right in the sales area behind the glass. Source: Walmart

Video surveillance in this case does not imply the identification of customers - the cameras monitor mainly the actions of customers, as well as the goods on the shelves. In the dark corners (say, in the depths of the shelves) there are sensors that help the system understand if the goods are in place or not. In addition, cameras can recognize the gender, age, and type of the client’s figure and display digital signage for him on which he passes, special offers and advertisements corresponding to these characteristics.

What can be shopping under the camera? Kindly: once you went to the store, and “smart” cameras have already determined the gender, age, and even assumed idle status. You head to the drinks department for juice, but here you come across toys and stick a little on the radio-controlled cars, forgetting where you went. Next month, on the recommendation of AI, the departments will switch places, because you always buy juices and the like for you, and never toys. You get to the shelves with juice and, finding that your beloved apple is not there, you are going to leave, but here you see a store employee who rolls up a cart with your favorite juice. Cameras and sensors found a lack of this product on the shelf and let them know in the warehouse.


Toshiba's smart video surveillance systems can capture slower customer walks in shop windows that interest him. This information can be used to further optimize the placement of various stores in shopping centers. Source: Toshiba

Now it's time for the sausage - you already reached for your favorite Krakow, but here again the store employee intervenes in your peaceful shopping. He takes the sausage off the shelf because its expiration date was determined by its color and shape of the chamber. You take a doctorate, move on, and then a store employee brings a basket that you forgot to take at the entrance: the cameras noticed that your hands were busy with beer and sausage, and sent a signal to the staff. Moreover, it was they who gave you the basket, not the cart, because the cameras saw how you came on foot and did not arrive by car, from which we concluded that you do not need a lot of products. You are about to leave, but you see a strange person - he looks suspiciously around and looks at a bottle of good whiskey for 100,500 rubles. This is the same stranger who hung out at the neighboring Bentley in the morning. But then a guard approaches him and begins a conversation with him. The cameras recognized the suspicious behavior characteristic of petty thieves and alerted personnel before the crime occurred.

To date, far from all of the described features have been implemented. The Walmart “smart” hypermarket mentioned above is still a single example, and retailers use “smart” cameras primarily to analyze consumer traffic to optimize their business. Such solutions are already supplied by companies such as GoodVision. Theft prediction cameras have been developed (Japanese startup Vaak), but not yet widely implemented. Finally, the technology of visual quality control of the product and its assortment is a matter of the future, since it is still more profitable for retailers to attract relatively inexpensive labor for these purposes.

Afterword: Big Brother - good or evil?

In May 2019, San Francisco authorities prohibited police and other municipal services from using smart cameras with face recognition. Innovation-friendly San Francisco was the first city in the United States to impose such a ban. The city council decided that this technology threatens the rights of the population. Meanwhile, in another American city, Boston, it was these systems that helped to find the perpetrators of the attack at the marathon two years earlier, and there is no ban there. Who is right? Perhaps the one who understands that the technique is impartial, and only the end user of its capabilities is important. It is his motives that determine whether the lens of a smart camera is aimed at us, or whether he only carefully watches us.