The sinister secret in the heart of artificial intelligence

No one understands how the most advanced algorithms work. And that could be a problem

Last year, a weird robomobile stepped onto the quiet roads of Monmouth County, New Jersey. The experimental vehicle developed by researchers from Nvidia was not outwardly different from other robomobiles, but it was completely different from that developed by Google, Tesla or General Motors, and it demonstrated the growing power of AI. The car did not follow the strict instructions programmed by the person. He relied entirely on an algorithm that had learned to drive a car while observing people.

To create a robomobile in this way is an unusual achievement. But also a little alarming, because it is not completely clear how the machine makes decisions. Information from the sensors goes directly to a large network of artificial neurons that processes the data and issues the commands necessary to control the steering wheel, brakes and other systems. The result is similar to the actions of a live driver. But what if one day she does something unexpected - moves into a tree, or stops at a green light? In the current situation it will be very difficult to find out the reason for this behavior. The system is so complex that even the engineers who developed it can hardly find the reason for any particular action. And she cannot be asked a question - there is no easy way to develop a system that can explain her actions.

The mysterious mind of this car indicates an AI problem. The underlying AI technology, deep learning (GO), in recent years has proved its ability to solve very complex problems, and it is used for tasks such as creating image captions, voice recognition, text translation. It is hoped that such technologies will help in diagnosing deadly diseases , making multi-million dollar decisions in financial markets and in countless other things that can transform industries.

But this will not happen - or should not happen - unless we find a way to make technologies like GO more understandable for our creators and responsible for our users. Otherwise, it will be very difficult to predict the appearance of failure - and failures will inevitably occur. This is one of the reasons why Nvidia's car is in the experimental phase.

Already today, mathematical models are used as an auxiliary tool to determine who can be released on parole, whom to approve a loan for and who to hire. If you could access these models, you could understand how they make decisions. But banks, military, employers, and others are starting to pay attention to more complex machine learning algorithms that can make automatic decision making inexplicable. GO, the most popular of these approaches, is a fundamentally different way of programming computers. “This problem already matters, and in the future its value will only increase,” says Tommi Jaakkola, a professor at MIT who works on machine learning (MO) applications. “Is this decision connected with investments, with medicine,

Some already argue that the ability to interrogate the AI ​​system for how a particular decision was made is a fundamental legal right. Since the summer of 2018, the European Union may introduce a requirement according to which companies should be able to explain to users the decisions made by automatic systems. And this may not be possible, even in the case of systems that at first glance look simple - for example, for applications or sites that use GO to display ads or recommend songs. The computers on which these services work, programmed themselves, and this process is not clear to us. Even the engineers who created these applications cannot fully explain their behavior.

This raises difficult questions. With the development of technology, we may have to go beyond a certain limit beyond which the use of AI requires a certain belief in it. Of course, people cannot always fully explain the course of their thoughts - but we find ways to intuitively trust and verify people. Will this be possible with machines that think and make decisions differently than a person would? We have never created machines that work in ways incomprehensible to their creators. What can be expected from communication and life with machines that can be unpredictable and inexplicable? These questions have led me to the forefront of research on AI algorithms, from Google to Apple, and to many places between them, including meeting with one of the greatest philosophers of our time.



In 2015, researchers from Mount Sinai Medical Complex in New York decided to apply the GO to an extensive database of case histories. They contain hundreds of variables obtained from analyzes, visits to doctors, etc. As a result, the program, called the Deep Patient researchers, trained on the data of 700,000 people, and then, when tested on new patients, it showed surprisingly good results in predicting diseases. Without the intervention of experts, Deep Patient found patterns hidden in the data, which, apparently, indicated that the patient had embarked on a path to various diseases, including liver cancer. There are many methods to “predict well enough” based on a case history, said Joel Dudley, a team leader. But, he adds, “this one just turned out to be much better.”

At the same time, Deep Patient is puzzling. She seems to well recognize the initial stages of mental abnormalities like schizophrenia. But since it’s very difficult for doctors to predict schizophrenia, Dudley wondered how it works in a car. And he still has not been able to find out. The new tool does not provide an understanding of how it achieves this. If a system such as Deep Patient ever helps doctors, ideally it should give them a rationale for their prediction to convince them of its accuracy and justify, for example, a change in the course of medication. “We can build these models,” Dudley states sadly, “but we don’t know how they work.”

AI was not always like that. At first there were two opinions about how AI should be understood or explainable. Many believed that it makes sense to create machines that reason according to the rules and logic, making their internal work transparent to anyone who wants to study them. Others believed that intelligence in machines could arise faster if inspired by biology, and if the machine learns through observation and experience. And this meant that you need to turn all programming upside down. Instead of the programmer writing down commands for solving the problem, the program would create its own algorithms based on sample data and the desired result. MO technologies, which today have turned into the most powerful AI systems, have taken the second path: the machine programs itself.

At first, this approach was of little practical use, and in 1960-70 he lived only at the forefront of research. And then the computerization of many industries and the emergence of large data sets returned interest in him. As a result, the development of more powerful machine learning technologies, especially new versions of artificial neural networks, began. By the 1990s, neural networks could automatically recognize handwritten text.

But only at the beginning of the current decade, after several ingenious tweaks and corrections, deep neural networks showed a dramatic improvement in performance. GO is responsible for today's AI explosion. It gave computers extraordinary capabilities, such as speech recognition at the human level, which would be too difficult to program manually. Deep learning has transformed computer vision and radically improved machine translation. Now it is used to help make key decisions in medicine, finance, manufacturing - and much more.



The operation scheme of any MO technology is inherently less transparent, even for computer scientists, than a manually programmed system. This does not mean that all AIs in the future will be equally unknowable. But essentially GO is a particularly dark black box.

You can’t just look into a deep neural network and understand how it works. The reasoning of the network is embedded in thousands of artificial neurons organized in dozens or even hundreds of complexly connected layers. The neurons of the first layer receive input data, such as the brightness of a pixel in the picture, and calculate a new output signal. These signals are transmitted along a complex web to neurons of the next layer, and so on, until the data is completely processed. There is also a backpropagation process that adjusts the calculations of individual neurons so that the network learns to produce the necessary data.

Many layers of the network allow it to recognize things at different levels of abstraction. For example, in a system configured for dog recognition, lower levels recognize simple things, such as a path or color. Higher ones already recognize fur or eyes. And the top ones identify the dog as a whole. The same approach can be applied to other input options that allow the machine to train itself: sounds that make up words in speech, letters and words that make up sentences, or steering movements necessary for driving.

In an attempt to recognize and explain what is happening inside the systems, inventive strategies have been developed. In 2015, Google researchers changed the image recognition algorithm so that instead of finding objects in the photo, he would create or modify them. In fact, by running the algorithm in the opposite direction, they decided to find out what features the program uses to recognize, say, birds or buildings. The final images created by the Deep Dream project showed grotesque, alien animals appearing among clouds and plants, and hallucinogenic pagodas visible in forests and mountains. Images proved that civil defense is not completely unknowable. They showed that the algorithms aimed at familiar visuals such as beak or bird feathers. But these images also told how much different is the perception of a computer from a human, because a computer could make an artifact from something that a person would ignore. Researchers noted that when the algorithm created an image of a dumbbell, together with it he painted a human hand. The machine decided that the hand was part of the dumbbell.

Further, the process moved thanks to ideas borrowed from neurobiology and cognitive science. A team led by Jeff Clune, an assistant professor at the University of Wyoming, tested deep neural networks using the equivalent of optical illusions. In 2015, Klun’s group showed how certain images can trick the network into recognizing objects that weren’t in the image. To do this, low-level parts that the neural network is looking for were used. One of the members of the group created an instrument whose operation resembles an electrode implanted in the brain. He works with one neuron from the center of the network, and searches for an image that activates this neuron more than others. Pictures are abstract, demonstrating the mysterious nature of machine perception.

But we are not enough only hints at the principle of thinking of AI, and there is no simple solution. The interconnection of calculations within the network is critical for recognizing high-level patterns and making complex decisions, but these calculations are a quagmire of mathematical functions and variables. “If you had a very small neural network, you could understand it,” says Yakkola, “but when it grows to thousands of neurons per layer and hundreds of layers, it becomes unrecognizable.”

Next to Yakkola in the office is the workplace of Regina Barzilay, a professor at MIT who intends to apply MO to medicine. A couple of years ago, at the age of 43, she was diagnosed with breast cancer. The diagnosis was shocking on its own, but Barzilai was also worried about the fact that advanced statistical methods and MO were not used for cancer research or for developing treatment. She says that AI has great potential for organizing a revolution in medicine, but its understanding extends beyond the simple processing of medical records. She imagines the use of raw data that is not used today: “images, pathology, all this information”.

At the end of cancer-related procedures, last year Barzilai and students began working with doctors at the Massachusetts Hospital to develop a system that can process pathology reports and identify patients with specific clinical characteristics that researchers would like to study. However, Barzilai understands that the system must be able to explain the decisions made. Therefore, she added an additional step: the system extracts and highlights parts of the text that are typical of the pattern found to her. Barzilai and students are also developing a deep learning algorithm that can detect early signs of breast cancer in mammograms, and they also want to make this system able to explain their actions. “We really need a process in which the machine and people can work together,” says Barzilai.

The U.S. military is spending billions on projects that use the MoD to pilot cars and planes, set goals, and help analysts filter out huge heaps of intelligence. Here, the secrets of the operation of algorithms are even less relevant than in medicine, and the Ministry of Defense has defined explainability as a key factor.

David Gunning, director of development at the Advanced Defense Research Agency, oversees the Explainable Artificial Intelligence project (explained by AI). A gray-haired veteran of the agency, who previously followed the DARPA project, which essentially led to Siri, Gunning says automation is sneaking into countless military areas. Analysts are testing the ability of the MO to recognize patterns in huge volumes of intelligence. Autonomous cars and aircraft are developed and tested. But soldiers are unlikely to feel comfortable in an automatic tank that does not explain their actions to them, and analysts will be reluctant to use the information without explanation. “In the nature of these MO systems, it is often a false alarm, so the analyst needs help to figure out

In March, DARPA chose to fund 13 scientific and commercial projects under the Gunning program. Some of these may be based on the work of Carlos Guestrin, a professor at the University of Washington. He and colleagues developed a way in which MO systems can explain their output. In fact, the computer finds several examples of data from the set and provides them as an explanation. A system designed to search for terrorist emails can use millions of messages to train. But thanks to the approach of the Washington team, it can highlight certain keywords found in the message. Guestrin's group also came up with how image recognition systems could allude to their logic, highlighting the most important parts of the image.

One drawback of this approach and others like it is the simplified nature of the explanations, which may cause some important information to be lost. “We have not reached the dream in which AI can lead a discussion with you and is able to explain something to you,” says Goustrin. “We are still very far from creating a fully interpretable AI.”

And we are not necessarily talking about such a critical situation as cancer diagnosis or military maneuvers. Knowing the progress of AI discussions will be important if this technology becomes a common and useful part of our daily lives. Tom Gruber, Apple's Siri development management team, says explainability is a key parameter for their team trying to make Siri a smarter and more capable virtual assistant. Gruber did not talk about specific plans for Siri, but it is easy to imagine that when you received a restaurant recommendation, you would like to know why it was made. Ruslan Salakhutdinov, director of AI research at Apple and associate professor at Carnegie Malon University, sees explainability as the core of evolving relationships between people and smart machines. “She will bring trust to the relationship,” he says.

Just as it is impossible to explain in detail many aspects of human behavior, it is possible that AI will not be able to explain everything that it does. “Even if someone can give you a logical explanation of their actions, it will still not be complete - the same is true for AI,” says Klyun from the University of Wyoming. “This feature can be part of the nature of intelligence - that only a part of it can be rationally explained. Something works on instincts, in the subconscious. "

If so, then at some point we will just have to believe the decisions of AI or do without them. And these decisions will have to affect social intelligence.. Just as society is built on contracts related to expected behavior, so AI systems must respect us and fit into our social norms. If we create automatic tanks and robots for killing, it is important that their decision-making process matches our ethics.

To test these metaphysical concepts, I went to Tufts University to meet with Daniel Dennett, a famous philosopher and cognitive scientist studying consciousness and reason. One of the chapters of his last book, From Bacteria to Bach and Beyond, an encyclopedic treatise on the subject of consciousness, suggests that the natural part of the evolution of intelligence is the consciousness of systems that can perform tasks that are inaccessible to their creators. “The question is, how do we prepare for the wise use of such systems — what standards do we need from them and from ourselves?” He told me in the middle of a mess in his office on the university’s idyllic campus.

He also wanted to warn us about the search for explainability. “I think that if we use these systems and rely on them, then, of course, we need to be very strict about how and why they give us their answers,” he says. But since there may not be an ideal answer, we must be as careful about the explanations of AI as our own - no matter how smart the machine seems. “If she can’t better explain to us what she’s doing,” he says, “it’s better not to trust her.”