# A short guide on creating oracles, car gods and second-order errors

Probably, in this text for many there will be no novelty. Probably others will say that this does not happen in real life. But, not the first of April, but everything written here is the pure truth that happened to me or to people around. Perhaps some of the above will make you rethink the phenomena surrounding you.

If we approach these stories formally, we can say that they are all generated by the fact that people do not take into account the error of the second kind. At Yudkovsky, with whom a quarter of Habr is familiar - this error is usually called " Confirming distortion ."

What it is? In a nutshell - "a person is looking for confirmation of his model, not its refutation." The only chance to explain better is examples, examples, examples and experience. This is the only way to develop the feeling that "something is wrong here."

It seems to me that this short story will allow you to look at errors of the second kind from a completely different perspective. From the side of how they have already entered our lives, they affect almost every decision. And they help us make gods from surrounding technologies. In machine learning, I come across this every day.

## Introduction

Once, back in the days of the Union, a council meeting of several design bureaus was held. One of the doctors of science, the head of a large department in one of the design bureaus, presented a new method for finding flying objects by radar. At the same time, the accuracy of the finding was almost perfect, superior to existing algorithms by orders of magnitude.

It was based on some tricky fractal math. No one on the board could understand her from the report / accompanying papers. And the person who presented the algorithm gave more and more arguments that all radars should be converted to this algorithm. And his reputation was well-established. Everyone felt the catch. But no one could understand where the mistake was in mathematics.
And only one graduate student on the council, who got there by chance, replacing his boss, after an hour of discussion, decided to ask: "And what is the probability of false detection if there are no objects?" He did not understand mathematics, but clearly felt that he could not jump an order of magnitude accuracy.

Everyone was silent. Everyone was sure that such a trivial thing was obviously verified. But the doctor, who was defending mathematics, turned pale, looked haggard, and only managed to say “We forgot to see ...”.

People, even smart and titled ones, first of all seek confirmation of their theory. And the more they are confident in it, the more evil a joke it can play with them.

## Feel on yourself

If the article continues further, then there will be no pictures! But I know that articles on Habré are opened only for the sake of best pictures!

And there are a lot of them on this topic. They confirm that the feature of trusting the simplest solution is inherent in us at the level of nature. All of you have seen them many times. And, probably, there is no need to explain:

Classic yellow (blue?) Dress.

Necker’s cube , as here without it.

Rotation to the right / left. The

peculiarity of these illusions is that they have a double interpretation. The brain goes in cycles only on one point of view which solves a problem. And overlooks other points of view. And just tell me that you can keep in mind two views at the same time !!!

It is like a religion that can explain the nature of phenomena. Like horoscopes that are about nothing, but a person seeks confirmation. Often you can’t determine what a smart system has counted for you and how to compare it with reality. Do not believe?

## Example number two, about crooks

Well, how do you like this project ? Using Neurosky to assess a child’s talents? Anyone more or less knowledgeable in technology understands that this is a wiring. The noisy testimony of Neurosky barely makes it possible to distinguish playing the guitar from reading a book.

And this is not to mention that everything is 100 times more complicated for children. Children are troubled, which generates a high level of noise. And, of course, the developers do not give any evidence of the algorithm and statistical significance. It would work - it would be a breakthrough. But this is just a way of wiring.

After my articleAbout NeuroSky, five people wrote to me with similar ideas in the mail / PM. Someone was just crazy, and someone quite cynically hinted that “well, you do the system, because it will show something anyway, it cannot give complete nonsense after training”. And the site that I cited a little higher was cited as an example.

And why are you not horoscopes?

Why is it here? Well, there is one office that deals with wiring. Well, there are Scientologists . After all, this is not massive!

But, it seems to me that not a couple. Such things fill daily life. And technology generates them every day. Below in the text there will be some more illustrative examples about which you all know very well (will you have time to remember before you read it? ). And in the comments you can bring your own.

The key to me in this case is that it shows that many people want to believe in technology / methodology. And it is one thing when people who do not understand it believe. And the second thing is the self-deception of the creators. The second, unfortunately, is progressing.

## Example Number Three, Everyday

It seems to me that stories should not be repeated, that each of them should show human nature from some other side. So I’ll move on to the other side.
Car plate numbers!

What ?? What are the mistakes of the second kind !? What kind of magic?

And they appear here very cool. Here's what you say, what is the number of the car: Is it

“M” or “H”? Is the region “71” or “21”?

And, nevertheless, on the basis of single images of such a plan, people begin to evaluate the quality of the system. Why do you think so? Usually because a person saw the number of such a car. And it is difficult for him to admit that in such a photo he may have a different interpretation.

There is a collapse of consciousness in favor of the nearest decision that is known. I gave the most revealing example, but if you are suddenly interested, herehere I’ll make out a few more.

And to convince him that the number could be different is often almost impossible. It’s easier to say “this is included in the percentage of algorithm errors”.

This is a very common trap of consciousness when working with images and recognition systems. Man is waiting for a ready-made solution. And all that came to another decision is wrong. The path is not interesting. Only the result is interesting. A couple of times a month I have to explain that a trained algorithm is not required to work on frames with noise.

I’m scared to think how some companies offer such a plan. Even in very good shots, it’s hard to find glasses / gloves and track their owner. Even if you are not an algorithm, but a living person.

At the same time, engineers at technical enterprises are not those people who are asked by the philosophical task "is it possible to understand from this video whether a person has safety glasses or not."

And yet many people believe that unreadable numbers are also well recognized. And the resolution can be increased hundreds of times. On a habr there was a wonderful article on this subject. And sometimes they are very offended that this is not so. Magic from the car does not work;)

## Example number four, or how you can be at the epicenter

I have given three different options. And everything can be answered with the standard: “oh well, it’s just incompetent people.” Or: "people are mistaken, with whom it does not happen."

But, such errors are much more common than you think. As soon as we deal with ambiguous decisions, with situations where 3-4 outcomes are possible, it is impossible to assess the level of our competence. Even if you are a very cool engineer.

I will tell you an epic story about how experts can make mistakes (or not make mistakes) when they find a god from a car.

Once they wrote to me with a request to arrange a consultation on the recognition of some medical tests. On the phone, I didn’t really understand the manager’s explanation of what should be recognized. Moreover, according to him, the case was about some “completely new” methodology. So we agreed to meet with the doctors who developed it (Skype for doctors is too complicated).

The next day, after two hours of meeting, my brain began to boil. I did not understand with whom I talked for two hours. With absolute crooks, with future Nobel laureates, or with people who are crazy.

A group of several doctors invented a universal test that revealed dozens, if not hundreds of diseases. By urine, by blood, by tears, and by saliva.

The method was the same: “mix the liquid with the reagent (one and the same), see how it dries and crackes”. The diagnosis was made according to how cracks will look, to what color this whole thing will be, to what form crystals will grow on a dried drop, how everything looks in polarized light.

And I must say that the first part, which was in the urine, was reasonably well founded.

When in the following days I tried to dig up information on the topic and interviewed doctors - they confirmed that a lot of these are very strongly distorted standard tests that take place in reality. Although somewhere about a third of the book shook their heads doubtfully.

But the second part, by blood, was a product of insanity. What is Theranos?! It was just that it was necessary not to allocate 4.5 billion to an American there, but to these comrades.

Researchers told me how, by a drop of blood, they can deliver cancer in the very early stages. Just using a microscope, a couple of drops of reagents from any laboratory (if memory serves, it was albumin). And they told how they successfully apply these methodologies in clinical practice for treating patients of a medical center (sic!). Here is a short list of what they detected: (Clickable)

In my somewhat awesome look and question about the evidence base, they told me about hundreds of articles in the literature, about two departments in Russia working on this technique and developing it. About a dozen candidates and doctors of science who defended themselves on the topic. And in parting, they presented two books on the topic where the preface was written by some academician:

Is this all true? I do not know. I realized that I lacked an understanding of how chemistry and biology work. No understanding of how to find this information. And on formal grounds. Well yes. There are a bunch of doctors and related articles. None of them have been published in a peer-reviewed international journal with a high level of citation. There are academics who talk about the successes of the methodology. But they, excuse me, are academicians of the RAMS. There is not a single criticizing article, only articles reporting on successes (Google google in 2016, since then something new could come out).

No, well, of course, I found the e-mail of the chairman of the pseudoscience commission, and some of his deputies. And he wrote about everything about it ... I didn’t receive an answer :)

And in person. I understand that these people believe in it. Damn, Russian medicine is not the place where you can do something on a hoax of this level. You need to really believe in your business.
But I understand that with the bases on which they measure, with the parameters that are there - at least half of what is said is doubtful. But something can really work.

That work, thank God I did not go. Perhaps because I said that in order to automatically recognize something, you must first collect a large base and confirm with an independent markup that it exists. Or maybe the truth is due to bureaucratic formalities.

Who is right here? Doctors who come up with a theory and believe in it? Or am I a person from the side to whom she seems insanity? I dont know. If the doctors are mistaken, then this is a classic mistake “lack of verification of counterarguments”. Or maybe I. And then it's almost the same :)

But since then I have two epic books and a phantosmogoric story.

## And a couple of examples about doctors ...

You know. Doctors are generally a fertile topic. When you try to recognize something by neural networks, you constantly come across this. I worked with a large number of radiographic examinations: fluorograms, mammograms, talked and looked at CT, saw different teeth.
And almost everywhere (in varying degrees) there is one and the same problem: different doctors read pictures in different ways. You ask two doctors to note the pathology in the pictures - and they have an intersection area of ​​20-30 percent. And each of them is confident in their decision, shows clear boundaries.

Justifies his answer. And again, the problem is that a person builds a good model in his head, projects it onto an image, and then searches for confirmation. But very often - this is the wrong answer. Or unreliable.

Unfortunately, the past centuries, medicine was not based more on “proof of each step by statistics”, but on a “logical explanation of each step”. It sits very deep in psychology. Sometimes it seems to me that a homeopathic doctor and a poor therapist work on the same principle. There is a set of basic attitudes and faith in it. Without analysis, where did what come from. Without awareness of what you can trust. Most likely, the set of attitudes of the therapist is closer to reality. But globally, neither one nor the other can prove those schemes that use. Maybe even both will appoint validol or send to drink motherwort.

Once I was asked to consult a surgeon ... The head of a department, a prominent specialist with 30 years of practice. Uncle told me how with 100% accuracy according to the questionnaire level “date of birth, date of first month, number of children, ..” out of 30 questions you can predict when there will be breast cancer. Well, that is, of course, ridiculous. But he suggests filling out this questionnaire to all his patients, drawing conclusions based on it.

## Examples of what we are fooled every day

Want more? .. Want something that everyone knows about? Well, take the great and terrible car that everyone is afraid of. Polygraph . What, do you still believe that it works? Link leads to wikipedia. Go through and watch the criticism. Why it does not work and what certainty there is. And still, many use them. Why are you not a god from a car?

Different companies regularly write to me that offer face recognition. They are very offended when I ask them to say what size their base is. They say that they have a cool model developed by the best physiologists.

One company with a chic office in the center of Moscow was especially remembered. Absolutely Inadequate NDAand insane pathos about the superiority of their technology ... Almost five years have passed since then. The company even had its own blog on Habré since then. Only here the products did not have anything to do with lie recognition, and they solve much more mundane tasks (as I understand it, they used the same developments, which is good).
It's funny that later it turned out that not only I sent them, but also other companies involved in the development of machine vision. Perhaps this allowed them to rethink what they were doing - and start doing business ^ _ ^

Finally, I can not help but give a link to the announcement of the Superjob portal that was a few days ago. I would like to offer them another 100% proven method to learn about the candidate more .

Perhaps the time has come to come to some conclusions? The conclusions that I made for myself - to start and do any work only after complete verification of the input data and methodology.

Do not trust anyone until he shows the verification of his method by open statistical research / access to the algorithm.

I am sure that these conclusions are not ideal. They are not suitable for medicine. Most likely there is no verified study of aspirin. For thousands of years, people have known that willow bark extract helps bring down temperature. Why research this?

And to summarize the findings and understand the universal strategy of trust vs. check - I can’t.

But probably if you start to wonder “is it possible to trust this or that method” - the article will already fulfill its goal. PS

Disclaimer

I deliberately tried not to mention the names of people / the names of companies where there is at least some chance that people are honestly mistaken. Or where can I be mistaken. Probably, according to what is in the text, many can be calculated or google. I would ask you not to post it in the comments. All who need it - do it yourself / will be able to ask in PM.

In many situations, I may not know the deep details of the solutions, so I do not consider it necessary to brand some people. But it seems to me important to show on these examples that our interaction with the modern world generates a bunch of errors of consciousness in people.