Statement of the problem of computer vision

For the last eight years I have been actively engaged in tasks related to pattern recognition, computer vision, and machine learning. It turned out to accumulate a sufficiently large baggage of experience and projects (something of their own, something in the rank of a full-time programmer, something for the order). In addition, since I wrote a couple of articles on Habré, readers often contact me, asking me to help with their task, to advise something. So quite often I come across completely unpredictable applications of CV algorithms.
But damn it, in 90% of cases I see the same system error. Every now and again. Over the past 5 years I have explained it to dozens of people. Why, periodically, I myself do it ...

In 99% of computer vision tasks, the idea of ​​the problem that you have formulated in your head, and even more so the solution path that you have outlined, has nothing to do with reality. There will always be situations that you could not even think of. The only way to formulate a problem is to build a database of examples and work with it, taking into account both ideal and worst situations. The wider the base, the more accurate the task. You cannot talk about a task without a base.

Trivial thought. But everyone is mistaken. Everything. In the article I will give some examples of such situations. When the task is bad, when it's good. And what pitfalls await you in the formation of TK for computer vision systems.

First examples

The bad

One of the most common ideas that I am asked about (even offered to take) is label recognition in stores: " Good afternoon! I came up with a cool startup: a person comes to the store, takes off the price tag, we find the product, price and look at which store the product is the cheapest! I’ve already done everything, but only the recognition module remains! " Over the past two years, with similar offers, they wrote me five times ...
And indeed! In the mind of a person who rarely encounters recognition tasks, there is a clear picture: “To recognize a line of text on a label is to spit it out!”. After all, there are ABBYY that recognize the text as pages, there are Smart Engines ( 1 , 2), for which cards with a bunch of numbers are recognized, and even labels! The problem has long been solved! What is the difference, Ikea or Auchan? All labels are similar, a single module will cope.
Usually after this one wants to tell a person: " Go to three different networks, take ten frames and look at them ." To which you can usually get the answer: " What I did not see there! Yesterday I was at the Crossroads and looked at them! ".
We will see? These are not the worst examples (photos are clickable). With all the examples here, a person can read / think through the information. What about the car? • Photos are often blurry, the text blurred and merged. Often the letters are written so close together that segmentation is almost impossible.



• There are a lot of artifacts around the edges of the price tags, often the letters are cut off, or there is a strip along them.
• If shooting with a flash, there will be glare, often completely covering the text.
• On the same price tag, often 2-3 prices are written in different fonts (and often the price tags can be emphasized to each other).
• The font changes even within the same distribution network.
• The format of price tags varies even within the same distribution network.
Some, especially stubborn, continue to insist: “You came up with everything! Here is a post at Smart Engines, where everything works and price tags are recognized! ”
And really! A wonderful example of a correctly posed problem: a rectangle of a given size is searched, on a red background, the font is the same. Having defined the boundaries of the rectangle, you can segment the code already. There is a heuristic, but minimal: connect the three blocks in the picture arranged in a known order.
Yes: there will be overexposures, there will be glare, corners can be bent, someone scribbles his autograph on the price tag, and for someone the camera always gives blurry shots. But when you know the position of each digit, everything else is not so important. And in most cases, everything will work just fine.
NBI am not saying that the task of recognizing price tags is not solvable in the general case. Solvable. And today's progress is making this decision closer and closer. Google already recognizes house numbers. And ABBYY is customizable to any predefined text format. But the solution to this problem is on the border of modern technology, the solution will be imperfect, or will require a huge amount of time and money for development. Of course, you can make price recognition on price tags (without text) and such a system will work well on some price tags. And sometimes you can read the barcode (from the price tags in open format, the barcode is written on one). Often there are ways to cut corners and simplify the task.

Let's get away from the labels

You will say that these are examples from the air and that this does not happen? .. I will give an example that was even published on Habré: habrahabr.ru/post/265209 .
Before reading further, try to understand why the method will not work.
- For those who are too lazy to read. The author suggests putting a mark on a tree of three points. And he believes that the intersection of lines between points with tree rings will allow you to clearly mark and classify the log from the camera.

Here he gives such a beautiful picture. The method is immediately clear and understandable. Is not it?
- The author of the article himself approached me about a week before publication with the question of whether this would work. I said that most likely I won’t, gave some examples, and also said how to modify the algorithm to make it work. But he wrote the article in the vein that this is a working method. And no one in the top ten comments objected. Only two dislikes ... (Damn, I confess, one of them is me).
Let's try to figure it out. Firstly, what does the “annual ring” look like? We ask Yandex to give us a log:

Ideal. Beautiful rings! Just like in the picture above. Wait ... what is this?

Also rings ... But what if we have a little missed camera with sharpness?

Pancake. Half the rings are gone. And what if the evening grows and ISO rises?

Again…
Well, maybe not everything is so bad? We’ll come up with a criterion to select only large enough annual bands, we will generate several options for each tree. OK?

No, there are still cracks that can change the geometry and situations when there are almost no bands at all. And this is the first 20 perfect pictures from the issuance of Yandex. The conclusion suggests itself in five minutes. But there is a cool idea! Why watch pictures from the search? ..

The task itself, in my opinion, is more likely to be solved. If you take the marks as reference points and compare with the same methods that compare the eyes . But, again, until you test the database with at least a couple of hundred examples, you never know whether the work can be successfully completed. But for some reason, the author of the article did not like such an offer ... Sorry!
These are the two most meaningful and representative, in my opinion, examples. From them you can understand why you need to abstract from the idea and watch real shots.
A few more examples that I met, but in a nutshell. In all these examples, people did not have any pictures at the moment when they began to inquire about the feasibility of the problem:
1) number recognition in the marathon on T-shirts for the video stream (picture from Yandyksa)

XLI . While preparing an article I came across this . A very good example where all potential problems are visible. These are different fonts, this is an unstable background with shadows, this is blurring and jammed corners. And the most important thing. The customer offers an idealized base.. Shot with a good camera on a sunny day. Try to see the numbers of athletes on T-shirts by searching for the yandyks.
Hy.Hy A couple of hours before publication, the author of the order suddenly came to me himself with a proposal to take up work, which I refused :) Still, this is karma, add this to the article.

2) Text recognition on photos of phone screens

3) And, my favorite example. Letter to the post office:
“ we need a program in the commercial sector for recognizing images.
The algorithm is the following. The program operator sets the image of the item (s) in several angles, etc.
Then, when this or the most similar image of the item appears, the program performs the required / specified act.
of course I can’t tell you the details yet. "(spelling, punctuation saved)

Good ones

But it is not all that bad! The situation when the task is set perfectly occurs frequently. My favorite: “You need software to automatically count moose in the photo.
I’m sending an example of a photo with moose. ” Both photos are clickable. I still regret that this task has not grown together. First, he defended the candidate’s and was busy, and then the customer somehow lost his enthusiasm (or found other performers). In the formulation there is not the slightest interpretation of the solution. Only two things: “what needs to be done”, “input”. A lot of input. All.

Thought - Conclusion

The only way to set the task is to gain a base and determine the methodology of work on this base. What do you want to get? What are the limits of applicability of the algorithm? Without this, you will not only not be able to approach the task, you will not be able to pass it. Without a database, the customer will always be able to say, “Such and such a case does not work for you. But this is a critical situation! I won’t accept work without him. ”

How to form a base

Probably all this was a prequel to the article. This article begins here. The idea that in any CV and ML task needs a base for testing is obvious. But how to gain such a base? In my memory, three or four times the first dialed base went down the toilet. Sometimes the second one. Because it was unrepresentative. What is the difficulty?
You need to understand that "collecting the base" = "statement of the problem." The collected base should:
1. Reflect the problems of the task;
2. Reflect the conditions in which the task will be solved;
3. Formulate the task as such;
4. Bring the customer and contractor to a consensus on what has been done.

Season

A couple of years ago, a friend and I decided to create a system that could work on mobile phones and recognize car numbers. Something even happened, and we wrote a series of articles about it (http://habrahabr.ru/company/recognitor/). At that time, we were very sophisticated in CV systems. They knew that it was necessary to collect such a base so that it would be bad. To look at her and immediately understand all the problems. We have put together such a base: We

made an algorithm, and it even worked well. Gave 80-85% recognition of selected numbers.
Well, yes ... Only in the summer, when all the rooms became clean and the accuracy of the system dropped by 5 percent ...

Biometrics

In our life, we have worked quite a lot with biometrics ( 1 , 2 , 3 ). And, it seems, they stepped on all possible rakes when collecting biometric databases.
• The base must be assembled in different rooms. When the device for collecting the base is only with the developers, sooner or later it turns out that it is tied to an adjacent lamp.
• In biometric databases, you need to have 5-10 images for each person. And these 5-10 shots should be taken on different days, at different times of the day. Approaching a biometric scanner several times in a row, a person is scanned in the same way. Approaching on different days - in different ways. Some biometric characteristics may vary slightly during the day.
• A database compiled from developers is not representative. They are subconsciously read so that everything works ...
• Do you have a new scanner model? Are you sure it works with the old base?
Here are the eyes collected from different scanners. Different fields of work, different highlights, different shadows, different spatial resolutions, etc.

Base for neural networks and learning algorithms

If you use some kind of learning algorithm in your code, write it is gone. You need to form the basis for training with its consideration. Suppose you have two very different fonts in your recognition task. The first occurs in 90% of cases, the second in 10%. If you cut these two fonts in a given proportion and learn them using a single classifier, then with a high probability the letters of the first font will be recognized, but the letter of the second will not. For the neural network / SVM will find a local minimum not where 97% of the first font and 97% of the second are recognized, but where 99% of the first font and 0% of the second are recognized. There should be enough examples of each font in your database so that training does not go to another minimum.

How to create a base when working with a real customer

One of the non-trivial problems in collecting the database is who should do it. Customer or contractor. First, I will give some sad examples from life.

I hire you to solve my problem!

This is exactly the phrase I heard once. And damn it, you can’t argue. But only the base would have to be assembled at the factory, where no one would let us in. And even more so, he would not let us mount the equipment. The data given by the customer were useless: an object of a few pixels, a very noisy camera with impulse noise, which periodically twitches, from the strength of twenty test images. On suggestions to put a better camera, choose a better angle for shooting, make a database of at least a couple of hundred examples, the customer responded with a phrase from the header.

We don’t have time to do this!

Once the director of a very large company (about 100 staff + offices in many countries of the world) offered to talk. In the product that this company released, part of the functionality was implemented by very old and very simple algorithms. The director told us that he had long dreamed of modifying this functionality into modern algorithms. Even hired two different development teams. But it did not grow together. One team, according to him, was theorizing too much, and the second did not know any theory and did trivialism. We decided to give it a try.
The next day we were given access to a huge array of raw information. Strongly more than I would have been able to view in a year. After spending a couple of days analyzing the information, we were wary of asking: “What exactly do you need from the new algorithms?” We were called a dozen two situations when the current algorithms do not work. But in a couple of days I saw only one or two indicated situations. Looking at another packet of data, I was able to find another one. To the question: “what situations bother your customers in the first place?”, Neither the director nor his chief engineers could give an answer. They did not have such statistics.
We investigated the issue and proposed a solution algorithm that could automatically collect all possible situations. But we needed to help with two things. Firstly, to deploy information processing on the servers of the company itself (we did not have sufficient computing power or a sufficient channel to the place where the raw data was stored). It would take a week of work as the administrator of the company. And secondly, the representative of the company had to classify the information collected by importance and how it should be processed (this is another three days). By this time, we had already spent two or three weeks of our time analyzing the data, studying articles on topics and writing programs to collect information (no agreement was signed at this point, everything was done on a voluntary basis).
To which we were told: “We cannot distract anyone to this task. Understand yourself. ” On what we bowed and left.

The customer gives the base

There was another case. This time the customer is smaller. And the system that the customer deals with is scattered throughout the country. But the customer understands that we will not collect the base. And he is trying hard to collect the base. Collects. Very large and varied. And even assures that the base is representative. Getting started. We are almost completing the algorithm. Before the surrender, it turns out that the algorithm works on the assembled base. And we satisfy the terms of the contract. But the base was unrepresentative. There are no 2/3 situations in it. And those situations that are present are disproportionate. And on real data, the system works much worse.
So it turns out. We tried our best. All that they promised was done, although the task was much more difficult than planned. The customer tried. Spent a lot of time collecting the base.
But the final result is worthless. I had to come up with something on the go, at least somehow plug holes ...

So who should form the base?

The problem is that very often computer vision problems arise in complex systems. Systems that have been made by many people for decades. And to understand such a system is often much longer than to solve the problem itself. And the customer wants the development to begin tomorrow. And of course, the proposal to pay for the preparation of TK and the base is 2 times the cost of the task, increase the time by 3 times, give access to their systems and algorithms, select an employee who will show and tell everything, is puzzling to him.
In my opinion, the solution of any computer vision problem requires a constant dialogue between the customer and the contractor, as well as the desire of the customer to formulate the problem. The contractor does not see all the nuances of the customer’s business, does not know the system from the inside. I have never seen an approach: “here is the money for you, make me a decision tomorrow” worked. There was a solution. But did it work as it should?
I'm trying to shy away from such contracts like fire. Whether I work myself, or in some company that took an order for development.
In general, the situation can be represented as follows: suppose you want to arrange your wedding. You can:
• Think through and organize everything yourself from start to finish. In essence, this option is "solve the problem yourself."
• Think through everything from start to finish. Write all the scripts. And hire performers for each role. Toastmaster so that guests do not get bored, a restaurant so that everyone cooks and holds. Write the basic outline for the host, the menu for the restaurant. This option is a dialogue. Provide artist data, paint everything that is required.
• Can be thought out in large blocks without going into details. To hire a host, let him do what he does. Do not coordinate the restaurant menu. Order a designer a selection of dresses, hairstyles, image. There’s a minimum of headache, but when stripping competitions begin, you can understand that something was done wrong. It is far from a fact that, having formulated the task in the style of “Recognize the Symbol for Me”, the contractor and the customer will understand the same thing.
• And you can order everything to a wedding agency. Expensive, no thinking at all. But no one knows what will turn out. Option - "do me good." Most likely, the quality will depend on the cost. But not necessarily

Are there any tasks where the base is not needed

There is. Firstly, in tasks where the base is too complicated. For example, the development of a robot that analyzes the video and makes decisions on it. Need some kind of test bench. You can make a base for some separate functions. But to make a base for a full cycle of actions is often impossible. Secondly, when research is in progress. For example, not only algorithms are being developed, but also devices that will be used to dial the base. Every day a new device, new options. When the algorithm changes three times a day. In such conditions, the base is useless. You can create some kind of local database, changing every day. But something global is meaningless.
Thirdly, these are tasks where you can make a model. Modeling is generally a very large and complex topic. If it is possible to make a good model cheaply, then of course you need to do it. If you want to recognize text where there is only one font, it is easiest to create a modeling algorithm ( an example of such a task ).

Scientific approach

But what about scientists? Is it possible for each work they collect a separate base?
Usually not. On the Internet you can find a lot of open databases. Usually universal, for some classic examples. For example, there are several sites with bases for biometrics (the most famous ). There are sites with bases for testing various training algorithms ( 1 2 3 ).
The problem with all of these bases is often that they are of little use and are not representative. Take for example the legendary MNIST - a database of images of numbers with manual spelling:

All machine recognition algorithms are tested on it. Everything would be fine, but ... Top algorithms have long had an accuracy of the form 99.5%, 99.6%, 99.6351%, etc. 30-40 pictures that are well known to all are not recognized. It’s unrealistic for a person to recognize half of them. With tricky settings, you can slightly correct the accuracy and make + 0.1%. But it’s clear that neither of the real data, and even more of a qualitative evaluation of the algorithm, has anything to do with this.
Often it turns out that an algorithm written on such databases will work only under those conditions and with those parameters for which the entire database is assembled.

Give your examples!

There are many people on the hub who are engaged in image processing and probably have a lot of experience in this (I read some of them as a student as articles): SmartEngines sergeypid BelBES mephistopheies rocknrollnerd YUVladimir Nordavind BigObfuscator Vasyutka
(Sorry if I noted someone not on business, but most of those noted, they wrote cool articles on CV and ML). Surely you have your own thoughts on how to make the problem statement perfect and put together a cool base. Share it? Or maybe you will criticize what was written as heresy from beginning to end? :)