The philosopher of artificial intelligence Eliezer Yudkovsky about singularity, Bayesian brain and goblins in the closet

Eliezer Shlomo Yudkovsky is an American expert in artificial intelligence who studies the problems of technological singularity and advocates the creation of Friendly AI. In non-academic circles, he is best known as the author of the fanfiction of “Harry Potter and Rational Thinking” under the auspices of Less Wrong.

I have always been amazed by smart people who believe in things that seem absurd to me. For example, the geneticist and director of the National Institutes of Health, Francis Collins, believes that Jesus has risen from the dead. AI theorist Eliezer Yudkovsky believes that cars ... But I would rather give the floor to him. In 2008 I interviewed him on Bloggingheads. Tv, but nothing good came of it, as I decided that he was a follower of the singularity guru Ray Kurzweil. But Yudkovsky did not follow anyone and never attended college. He is a stubborn and original theorist of intelligence, both human and artificial. His work (for example, an essay that helped me understand, or gave the illusion of understanding, the Bayes theorem) exudes the arrogance of a self-taught, whose sharp edges were not polished by formal education - but this is part of his charm. Even when he annoys you, Yudkowski is funny, fresh, provocative. For details of his biography, see his personal website or the website of the Institute for the Study of Machine Intelligence, in the basis of which he participated. And read this interview with the bonus in the form of comments from his wife, Briena.

Horgan : When at the party they ask you what you do, what do you answer?

Yudkovsky : Depends on the event. “I am a specialist in decision making theory,” or “Co-founder of the Institute for the Study of Machine Intelligence,” or, if it is a different type of party, I talk about my artistic works.

X: What is your favorite AI movie and why?

J: The AI in films is horribly standard. Ex Machina came as close to exclusion from this rule as one would expect.

X: Is the college utility overrated?

YU:I would be surprised if its usefulness was underestimated, given the social requirements for completing it. As far as I know, there is no reason not to believe economists who believe that the college has become a kind of “prestigious product” and that attempts to increase the volume of student loans simply increased the cost of the college and the burden of student debts.

H: Why do you write artistic stories?

Y: To rephrase Wondermark comics : “At first, I tried not to do this, but it didn't work.”

In addition, serious literature conveys knowledge, and fiction - experience. If you want to understand the proofs of the Bayes formulaI can use charts. If you want to feel how it is to use Bayesian logic, I need to write a story in which the character does.

H: Are you religious in any sense?

U: No. When you make a mistake, you need to avoid the temptation to go into defense, try to find some point of view from which you are at least a little right. It is much wiser to say, “Oh,” admit that you were not even a little right, swallow the bitter pill whole and continue to live. That is how humanity should relate to religion.

H: If you were the “King of the World,” what would be on the top of your to-do list?

YU:I once wrote: “The test for libertarian works like this: Imagine that you got the power; What do you think about first of all - about the laws that you will accept, or about the laws that you will repeal? ”I am not 100% libertarian, because not all of my Wishlist are expressed in the abolition of laws and the loosening of restrictions. But I imagine how I would try to create a world in which some unemployed person could offer you a ride up to work, get $ 5 for a 20 minute drive, and nothing bad would happen to him because of this. He would not need to lose unemployment insurance, register a business permit, lose medical insurance, undergo an audit, ask a lawyer to certify the compliance of his work with the rules of the Office of Occupational Safety, etc. He would have just added $ 5.

I would try to return to the state in which it would be as easy to hire an employee as in 1900. Perhaps now there is a sense in some security measures, but I would try to create such security that does not hold back a person and does not produce securities as a result of a simple return of a person to the economy.

I would try to do everything that smart economists have long been shouting about, and that no state does. Replace investment taxes and profits with consumption taxes and real estate taxes. Replace the minimum wage with negative payroll taxes. Establish a nominal GDP targeting policy for central banks and stop supporting structures “too large to go bankrupt”. Require that the loser in court pay for court costs during patent proceedings [ following the so-called English rule - in contrast to US law, according to which each of the parties by default deals with its own costs - approx. trans.] and return the duration of copyright to 28 years. Remove obstacles to the construction of houses. Copy Singapore medical insurance. E-government in Estonia. Replace committees and complex decision-making processes with specific individuals who make publicly documented decisions and are responsible for this. Conduct controlled experiments with different options for managing countries and take into account their results. I can continue the list for hours.

All this may not matter in two hundred million years. But nominal assets derived from an economic boom can serve well as I try to figure out what we will do with artificial intelligence. The obvious thing is the Manhattan project somewhere on the island, based on competition between the largest hedge funds, in which people can explore the problem of generalized artificial intelligence without publishing the results of their work to automatically bring the world to the end. And unless we accept that I have magical abilities or a fundamentally non-overthrowed regime, I do not see how any law I would accept would have delayed the AI ​​approaching strongly enough on a planet where computers are common everywhere.

But all this can still be considered an impossible mental experiment and in real life the likelihood of such an experiment is zero.

H: What is so good about Bayes theorem?

Yu: Well, for example, it is very deep. Therefore, it is difficult to answer this question briefly.

I could answer that the Bayes theorem can be called the second law of thermodynamics for cognition. If you have concluded that the probability of some assumption is 99%, whether it is milk in the supermarket or the anthropogenic cause of global warming, then you have a combination of fairly good a priori probabilities and fairly reliable evidence. This is not a requirement of regulations, it is the law. Just as a car cannot drive without dissipating entropy, you cannot get a good image of the world without performing a process where there is a Bayesian structure somewhere inside, even if the probabilities are not used directly in the process.

Personally, I think that the most important thing Bayes can offer us is the existence of rules, iron laws that determine whether the way of thinking works to mark reality. Mormons are told that they learn the truth of the Book of Mormon through a burning sensation in the heart. Conservatively, we take the a priori probability.Mormon books as one in a billion. Then we estimate the likelihood that the Book of Mormon is not true, and anyone experienced a burning sensation in the heart after being told that this should be expected. If you understand the Bayesian formula, we will immediately understand that the small probability of proof is incommensurable with the small probability of the hypothesis that we are trying to prove with its help. You don’t even need to come up with specific figures to understand that they don’t converge - as Philip Tetlock discovered in his study of " super- predictors", they often knew the Bayes formula, but they rarely gave certain numbers. In a sense, it is harder to deceive you if you realize that there is some kind of mathematics with which you can accurately determine the strength of a proof and see if it is enough to overcome a small the likelihood of a hypothesis. You can't just invent something and believe it because it doesn't work that way

X: Does the Bayesian brain hypothesis impress you?

Yu: I think that some people arguing on this subject are talking about different things. , is the brain a Bayesian algorithm? Mom, it's like asking if the Honda Accord works on the Carnot heat engine. If one person says: “Every car is a thermodynamic process that requires fuel and dissipates parasitic heat,” and another person hears: “If you build a diagram of a Carnot cycle and show its mechanics, he must agree that it looks like the inside of a Honda Accord "Then violent disputes are inevitable.

Some people will be very happy to open the internal combustion engine, find cylinders in it and say: "I am sure that they convert heat into pressure and help move the car forward!" And they will be right, but other people will say: “You focus on the only component of a much larger set of car parts. The catalytic converter is also very important, and it is not on your Carnot cycle diagrams. And sometimes we have an air conditioner that works in exactly the opposite way to how you say the heat engine works. ”

I don’t think it would be surprising if I say that people who say down upon it: “You are clearly unfamiliar with modern cars; you need a whole set of different methods to build a motor, such as candles and catalytic converters, and not just these thermodynamic processes of yours, “miss out on a key level of abstraction.

But if you want to know whether the brain can be considered literally Bayesian, and not a device that performs cognitive work, whose nature we can understand using Bayesian methods, then I can answer your question: "No, of course." In this "engine" it is possible and there are several Bayesian "cylinders", but a lot there will look as strange as seat belts and air conditioner. But these additions will not change the fact that in order to properly define an apple based on sensory evidence, something must be done that can be interpreted as a result of induction, which can understand the concept of an apple and be updated on the basis of evidence that distinguishes apples from non-apples.

H: Is it possible to be too rational?

YU:You can get into the so-called. "The Valley of Bad Rationality." If before that you were irrational in a few things that balanced each other, then if you become rational, you may become worse than before. The more you become rational, the worse you can become if you choose the wrong direction to apply your skills.

But I would not recommend too much to take care of such an opportunity. In my opinion, people who talk about how to be smartly irrational, they are assholes. It is difficult to come up with a realistic, lifeless situation in which you can decide to be irrational, and the outcome of which is unknown to you. In real life, it's better to tell yourself the truth and not be smart.

It is possible that the ideal representative of Bayesian thinking is incompatible with an interesting and fun life. But this is clearly not such a big problem as our tendency to self-destruction.

H: How does your point of view on singularity differ from that of Kurzweil?

U:
• I don’t think Moore’s Law can be applied to AI. AI is a software problem.
• I do not think that the first superhuman intellect will appear from the merging of machines with people. A hundred years have passed since the advent of cars, and we are only now trying to make an exoskeleton for a horse, and an ordinary car is still faster.
• I do not think that the first strong AI will be based on algorithms from neurobiology, just as the planes were not based on birds.
• I don’t think that fusion of nano, info and biotechnology is possible, inevitable, well defined or necessary.
• I think that there were more changes from 1930 to 1970 than from 1970 to 2010.
• I think that in developed countries, productivity stagnates.
• I think the extrapolation of Moore's law to technological progress, which allegedly predicts everything that will happen after the appearance of AI is smarter than a person, is a very strange thing. AI smarter than man destroys all your graphics.
• Some analysts, for example, Illka ​​Tuomi , believe that Moore's law broke in the early 2000s. Not sure I can argue.
• The only technological threshold that interests me is where the AI ​​gains the ability to self-improve. We do not have a schedule going to this threshold, and it is unclear what it will be (although it should not greatly exceed the level of a person, because a person understands computer science), so that his attack cannot be predicted.
• I don’t think that the result of such progress will be good by default. I think it can be made good, but we will need to work hard on this, and key figures are not interested in this. To tell people that we are on a natural trajectory to great and wonderful times will be a lie.
• I think that “singularity” became a suitcase word with too many incompatible values ​​and details inside, so I stopped using it.

X:Do you have the chance to become a super-intelligent cyborg?

Yu: The law of probability conjunction says that P (A & B) <= P (A). The probability of simultaneous occurrence of events A and B is less than the probability of the occurrence of one event A. In experiments in which people consider that P (A & B)> P (A) for two events A and B, there is a " conjunction error"- for example, in 1982, experts from the International Congress of Predictions appointed a greater likelihood of the event" Russia invades Poland and disintegrates diplomatic ties with the USSR "than the likelihood of a separate event" disintegration of diplomatic ties with the USSR ", appointed by another group. Similarly, another The group assigned a greater likelihood to the California Earthquake event leading to a flood causing thousands of victims, while the other - the likelihood of the event "Somewhere in North America there is a flood with thousands of victims." Putting additional details into the story unambiguously makes it less likely, it makes it more believable. For me, understanding this fact is like a " bridge of donkeys"for serious futurism - the difference between the fact that you carefully weigh each individual assumption, and find out whether you can support this clarification independently of everyone else, and the fact that you are simply composing a wonderful and vivid story.

This is all I say in the context of the answer to the question: "Why do you drag a clarification like a cyborg to this? I don’t want to be a cyborg." You need to carefully add additional details to the statements.

X: Do you have a chance for immortality?

Yu: Literal? Literal immortality is difficult to achieve. To live s much longer than a few trillion years, it is necessary to revise the expected fate of the expanding universe. In order to live longer than googolplex, we need to make a mistake about the fundamentals of physical laws, and not just in details.

Even if some of the unusual reasoning turns out to be true and our Universe can generate daughter universes, this will not give us immortality. In order to live a lot more years and not repeat a googolplex, you will need computers with more elements than googol, and such a machine will not fit in the Hubble sphere .

And googolplex is not infinity. Paraphrasing Martin Gardner, the Graham number is still quite small, since most of the finite numbers are much larger than him. If you want to demolish the roof, read about the fast-growing hierarchyand infinity will still be longer. Only very strange and frightening anthropic theories will allow you to live long enough to watch the longest-working Turing machine stop with hundreds of conditions.

However, I don’t think that from an emotional point of view I would want to live long enough to see the hundredth number in the game “the hunt for a beaver-hard worker". I can somehow empathize with myself, who has lived a hundred years from now. That future one will be able to empathize with the future for myself in another hundred years. And perhaps somewhere in this sequence there will be someone who will face the prospect of ending their existence, and he can be very sad about this, but I am not sure that I can imagine this person. "I want to live another day. Tomorrow I will also want to live another day. Therefore, I want to live forever, proven by induction of positive integers numbers. ”Even my desire is long . of life in the universe is physically possible - is an abstraction generated by induction can not imagine itself through a trillion years.

X:I described the singularity as an escapist and pseudoscientific fantasy that distracts us from climate change, war, inequality and other serious problems. Why am I wrong?

Yu: Because you are trying to predict empirical facts through psychoanalysis. It will never work.

Suppose we live to see the emergence of an AI, smart enough to enable it to do the same work to improve AI, which people do. He can tune himself, program, invent new algorithms. Improve. What will happen next - he will become smarter, see even more opportunities for improvement, and quickly get to a very high level? Or nothing much will happen?

It may happen that (A) self-improving a certain delta will make the AI ​​smart enough so that it can look back and find a new potential size improvement to * the delta, where k> 1, and this will be repeated many times to cause a quick self-improvement to level of superintelligence. What Irving John Good called the "explosion of intelligence." Or (B), to less than one, or all these improvements are small and do not lead to the emergence of superintelligence, or superintelligence is impossible at all, and instead of an explosion there will be puff. What is true, A or B? If you build an AI of a certain level and it tries to do it, something will happen in the empirical real world, and this event will be determined by facts concerning the landscape of the algorithms and the achievable improvements.

It is impossible to get reliable information about this event from the psychoanalysis of people. It’s like trying to start a car without fuel — something that Bayes theorem tells us. Some people will always be escapists, regardless of the real values ​​of the hidden variables in computer science, so observing certain escapists cannot be called rigorous evidence.

This is a misconception about the nature of rationality - that it is rational to believe that “goblins do not exist in cabinets” because belief in goblins from the closet is stupid, immature, outdated, and only idiots believe in it. The real principle of rationality is to go and check in the closet. So in those universes where goblins live in cabinets, you will believe in goblins, and in universes where goblins are not in cabinets, you will not believe in them.

It is difficult, but in principle, it is possible to try to peek through the open door and ask: “What would be different in the Universe if you could not get a good income from cognitive investments, that is, the AI, trying to improve itself, would end up with an explosion, and zilch? What other facts would be characteristic of such a universe? ”

There are people who claim that AI can only be raised to the human level, since we are human beings, and we cannot raise it higher. It seems to me that if our universe is like this, then we should observe a decrease in income from investments in hardware and software for computer chess that exceed the level of man — which is not happening. In addition, natural selection would not have been able to create a man, and Einstein’s mother should have been a terrific physicist, etc. etc.

There are people who argue that the more complex the algorithm, the more it needs adjustments, and that our intellect serves as a kind of restriction for this process. But this does not agree with the anthropological records of the human intellect; investments in brain and mutation adjustments enhance cognitive abilities. We know, because genetics tells us that mutations with a small statistical response are not fixed during evolution.

And hominids did not need an exponentially larger brain than chimpanzees. And the head of John von Neumann was not exponentially larger than the head of the average person.

From a purely practical point of view, human axons transmit information at a speed of a million times less than the speed of light, and even from the point of view of heat dissipation, each synaptic operation consumes a million times more than the minimum thermal dissipation of an irreversible binary operation at 300 Kelvin, and so on. Why should we assume that brain software is closer to optimum than iron? The privilege of human intelligence is that it is the lowest level of intelligence that can create a computer. If it were possible to create a computer with a lower level of intelligence, we would discuss it at a lower level.

But this is not a simple argument, and for a detailed description I send people to one of my old works, “Microeconomics of the Explosion of Intelligence,” which, unfortunately, still serves as the best source of information. But such questions need to be asked so that, using the available evidence, to reason about whether we will see an explosion of AI, in which some improvement in cognitive abilities, invested in self-optimization, will give an increase that exceeds this improvement.

Regarding the possibilities and their prices:

You can imagine a world without an explosion of intelligence and without superintelligence. Or a world where the tricks that machine learning experts will use to control super-AI are suitable for controlling people and the superhuman regime. Or a world where moral internalism workstherefore, all reasonably advanced AIs are good. In such worlds, all the work and all the anxieties of the Institute for Machine Learning Research are unnecessary. And a few mosquito nets were wasted, and it was better to give them to the fund to fight malaria.

You can also imagine a world in which you are fighting malaria, fighting and keeping carbon dioxide emissions at the proper level, or using geo-engineering solutions to neutralize the mistakes that have already been made. And all this turns out to be useless, since civilization is unable to solve the problem of the morality of AI - and all children saved from malaria with the help of nets grow only to nanomachines kill them in a dream.

I think that people trying to engage in reasonable charity would agree that we would not want to live in any of these worlds. The only question is which of them is more likely. The central principle of rationality is not to reject faith in goblins, because it is stupid and undervalued, and not to believe in goblins, because it is great and beautiful. The central principle of rationality - what observable signs and logical conclusions will help us choose one of these two worlds.

I think that the first world is unlikely, and the second is likely. I understand that trying to convince others of this is trying to swim against the current of faith in eternal normality. Belief in the fact that only our short-term civilization that has existed for several decades, and only our species, which exists only an instant on the evolutionary and geological scales, make sense and must exist forever. And although I believe that the first world is just an optimistic dream, I do not think that we should ignore the problem about which in the future we will panic. The mission of the Institute is to conduct research today, which, according to people living after 30 years, should have started 30 years ago.

H: Does your wife believe in singularity?

Briyenne: If someone asked me if I believed in singularity, I would raise an eyebrow and ask if they believed in automatic trucks. This is a strange question. I do not know what the first fleet of automatic trucks will be, or how much time they will need to replace the existing freight system. And I don’t really believe in the robotic trucks, I confidently predict that unmanned transportation will replace modern transportation with the participation of people, because it is in this direction that we are going if nothing really strange happens. For the same reason, I confidently predict an explosion of intelligence. In other meanings of the word "singularity" I am not so sure.

Yu: Briyena gave her answer without seeing my answers. Simply, we are well suited to each other.

X:Is it possible to create superintelligence without understanding how the brain works?

Yu: In the same sense that you can make airplanes, not understanding how a bird flies. You do not need to be an expert on birds, but at the same time for the construction of the aircraft you need a lot of knowledge, having obtained that, in principle, you can already understand how about a bird soars or repels from the air. Therefore, I am writing about human rationality - if one can go far enough in the issue of machine intelligence, one cannot help but think of some ideas about how people think.

H: What can superintelligence want? Will they have something like sexual desire?

YU:Think of a huge space of opportunity, a giant multidimensional sphere. This is the space of mind species, a set of all possible cognitive algorithms. Imagine that somewhere at the bottom of the sphere there is a tiny dot that indicates all people who have ever lived. This is a tiny point, because all people have approximately the same brain, with the cortex, cerebellum, thalamus, etc. Some people are not like others, so this may be a spiky point, but the spikes will be of the same scale as the point itself. Regardless of your neuroarity, you will not work on another cortical algorithm.

Asking “what superintelligence wants” is the wrong question. Superintelligence is not a strange tribe of people living on the other side of the river, and possessing exotic customs. AI is simply the name of the entire space of possibilities beyond a tiny human point. With sufficient knowledge, you can get into this space of opportunity and get an AI with desires that can be described by the human language in women, but not because they are natural Wishlocks of these exotic supermen, but because you selected one part of the mind species. .

Regarding sexual desires - if you know exactly what you are doing, you have solved the main problem of building an AI that stably wants certain things while it is improving, if you have solved the main problem of channeling the utilitarian functions of AI to tasks that seem deceptively simple to a person, and an even more difficult problem of building AI using a certain kind of architecture, in which such things as “sexual desires” and “happiness from sex” matter, then maybe you can make AI look at people who are modeling their desires, extract that part of them as regards sexual desire, and make the AI ​​experience it.

Of course, it is also possible, with good knowledge of organic biology and aerodynamics, to build airplanes that can mate with birds.

But I do not think that the Wright brothers should have done such things at the dawn of their work. That would not make sense.

It seems more reasonable to solve the problem of penetration into the space of minds and extract from there the AI, who does not want to disassemble us into spare parts-atoms.

H: I want to think that super-intelligent beings will practice non-violence, because they will understand that violence is stupid. Am I naive?

Yu: I think so. David Humewould tell you that you are making a typical mistake by applying the “stupidity” predicate to the values ​​or official operations of an individual. Actions, choices, rules can be stupid if you have any preferences about the final states of the world. If you are a person with meta-preferences that you have not fully computed, you may have a platform on which you can rely and call certain assumptions about object preferences "stupid."

Stapler maximizer [a thought experiment that demonstrates how an AI, made without malicious intent, can harm humanity - approx. trans.] does not make a computational error, choosing those cases from the possible in which the number of clips is maximized. He is not inside your platform of preferences, choosing erroneous actions, and he is not inside your platform of meta preferences, mistakenly choosing preferences. He calculates the answer to another question, and not the one that you ask yourself, the question "What should I do?". The clip maximizer simply performs the action leading to the largest number of clips.

The fatal scenario is when the AI ​​neither loves you nor hates you, because you are made of atoms that it can use to create something else. Game Theory and Problems of Cooperation in the Prisoners' Dilemmado not manifest themselves in all possible cases. For example, they do not appear when a certain subject is so much stronger than you that he can disassemble you into atoms, when you want to press the "cooperate" or "change" buttons. And when we pass this threshold, then either you have solved the problem of creating something that does not want to harm you, or you have already lost.

H: Will superintelligence solve the difficult problem of consciousness ?

Yu: Yes, and looking back, the answer will seem shamefully simple.

H: Will superintelligence have free will?

Yu: Yes, but they will not have the illusion of free will.

H: What does your utopia look like?

Yu: I will direct your readers to my " Theory of entertainment theory"because I have not yet managed to write a story, the action of which takes place in the theoretical and entertaining optimal world.