There is no fire alarm for AI

Why do we need a fire alarm?

You might decide that a fire alarm should give you important evidence of a fire, which allows you to change your plans and leave the building.

In a classic experiment conducted by Latane and Darley in 1968, eight groups of three students were asked to fill out a questionnaire, sitting in a room that soon after the start of the experiment began to fill with smoke. Five of the eight groups did not respond and did not complain about the smoke, even when it became so dense that they began to cough. Subsequent actions showed that a single student complained of smoke in 75% of cases. A student in the company of two other people who were asked not to do anything, reacted to smoke only in 19% of cases. This and other experiments have shown that pluralistic ignorance is actually occurring. We do not want to seem alarmist and be afraid of an emergency, so we try to look calm, watching the others behave out of the corner of the eye - but they, of course,

I read a large number of reports on the reproduction of this experiment and its variations, and the results were blatant. It seems to me that this result was not destined to die in a crisis of reproduction, and I did not hear about the problems with its reproduction.

Fire alarm creates a well-known knowledge of the presence of a fire; after that, it will be socially acceptable to react to what is happening. When the alarm goes off, you know that everyone else also knows that there is a fire, and you know that you will not be dishonored if you follow the exit.

Fire alarm does not tell us that there is definitely a fire. I can’t remember a single incident in my life when I left the building due to a fire alarm, and there really was a fire. In fact, the fire alarm is no longer talking about the presence of a fire, but about the presence of smoke coming from under the door.

But the fire alarm reports that now from a social point of view it is acceptable to react to a fire. She promises that no one is ashamed of us because we calmly headed for the exit from the building.

It seems to me that this is one of those cases when people do not really understand what they believe in, as in the case when someone loudly asserts about the victory of the team of their city in the competition, but immediately calms down if you offer him money to win. They do not share a consciously pleasant exciting feeling from cries for victory and a feeling of real waiting for the team to win.

I think that when people look at the smoke coming from under the door, they consider that their uncertainty about the presence of a fire arises from the fact that they do not assign a sufficiently high probability to this event, and that they fluctuate because they are afraid to spend time and effort. But in this case, they misinterpret their feelings. In this case, they would experience the same vibrations at the sound of a fire alarm, or even fluctuate even more - the fire alarm correlates not with a fire, but with smoke from under the door. In fact, hesitations arise from worries about what other people think differently, and not because there may not be a fire. Reluctance to act grows from unwillingness to seem like a fool, and not from unwillingness to waste energy.

* * *

From time to time we are offered to postpone reaction to the problems associated with general-purpose artificial intelligence (ION), since it is believed that we are still too far from its appearance, and now it is simply impossible to do something productive with it.

Some people still believe that today there are specific things that we can already do: Soares and Fallenstein (2014/2017); Amodei, Olah, Steinhardt, Christiano, Schulman, and Mané (2016); or Taylor, Yudkowsky, LaVictoire, and Critch (2016).)

Even if these works did not exist, or if you, as an AI researcher, read them, and decided that this is nonsense, and you would like to work on this problem, but you would not know, somehow your next wise step was would sit down and spend a couple of hours, sincerely trying to come up with possible solutions. It is desirable without self-sabotage, thinking that you still can’t think of anything - in principle, it’s much more convenient to decide that you can’t do anything, because you have things that you are more interested in doing. Anyway.

So if the idea of ​​ION seems too far away, and you think that it gives you the right to believe that you can’t do anything productive about organizing its work, then the following alternative strategy follows: sit and wait for future some unknown event so far tells us that ION is approaching, and then we will already understand that we can begin to work on organizing ION.

It seems to me that this is the wrong approach for a variety of reasons. And here are some of them.

The first. As Stewart Russell said, if you received radio signals from space, noticed a spaceship through a telescope and know that aliens will land in thirty years, you will still start thinking about it today.

You will not say: "Well, it will be in thirty years, and let him." You certainly will not say: “We can’t do anything until they come closer.” You will surely spend a couple of hours, or at least five minutes, brainstorm ideas, thinking about whether you can do something now.

If you say that aliens will arrive only after thirty years, and therefore today you have nothing to do ... In more efficient times, someone would ask you to provide a schedule of actions - when and what to do, and long before arrival. And if you would not have such a schedule, then the questioner would understand that you are not acting according to the working table of timely responses, but simply prokrastieniruete. From this, he would make the correct conclusion that you probably did not think well enough about what you can do today.

Using the expressions of Brian Kaplan [ American economist and blogger / approx. trans.], everyone who calmly believes that “now nothing can be done to prepare,” is set up incorrectly; they should be more concerned about the fact that they cannot come up with a single way to somehow prepare. Or maybe ask someone else if he has any ideas on this? Anyway.

The second. History shows that ordinary people, and even scientists who are not part of a key group of advanced researchers, and even scientists who are included there, often believe that key technological discoveries will not occur many decades later, although in reality they are five years from now.

In 1901, two years before he took part in building the first aircraft heavier than air, Wilbur Wright told his brother that even fifty years would pass before active flights.

In 1939, three years before he personally observed the first nuclear chain reaction in a handful of uranium blocks, Enrico Fermi expressed a 90% certainty that uranium could not be used to maintain the fission chain reaction. It seems to me that a year after that, that is, two years before the denouement, Fermi also said that if it is possible to receive decay energy, then it should be another fifty years.

Well, if you are not the Wright brothers and not Enrico Fermi, then you will be even more surprised. Most of the world learned that atomic weapons already exist and work only after reading the headlines about Hiroshima. Four years after the flight of the Wright brothers flyerthere were also respected intellectuals who claimed that the flight of vehicles is heavier than air is impossible, because then knowledge spread more slowly [ the first flight on an airplane is Clement Ader's flight , made in 1890, 13 years before the Wright brothers - so these intellectuals were 17 years late / approx. trans. ].

Can we say today, in hindsight, that then there were some events that signified the approach of flights heavier than air or nuclear energy? Of course, but if you go back to the past, read the newspapers of that time, and find out what people said about it then, it becomes clear that they did not know that these events were some kind of symbolic, or they very strongly doubted their significance. . Others, I think, played the role of enthusiastic futurists and proclaimed the inevitable onset of great change, while others - the role of skeptical scholars trying to stifle children's enthusiasm. And if among all this hype there would have been a predictor who said “decades” when it really took decades, and “5 years” when it really took five years - well, good searches for you in this noise. Probably,

One of the main reasons that we are strong in hindsight and believe that the past was more predictable than anyone could really predict at one time, is that we know in hindsight what we need to notice and focus only on one thought related to what exactly each of the available evidence means. But if you look at what people were saying at the time, you will see that they don’t even know what will happen in three months, since they don’t know which events were significant.

You can say that “until ION is 50 years old,” and this may be true. In the same way, people said that active flight would take place not earlier than in a few decades, and it took place in a few decades. The problem is, all the evidence looks the same to you, if you really live in this story, and do not read about it afterwards.

This does not mean that whenever someone says “after fifty years,” this event occurs after a couple of years. This means that a confident prediction about the remote onset of the event corresponds to the knowledge about the technology, which does not change until you get very close to the cutting edge of development. It's akin to “I don't know how to do it,” and sometimes you say that the event will happen in fifty years, and sometimes that in a couple of years, and sometimes you say it, while the Wright brothers flyer flies somewhere where you can not see it.

Third. Progress moves advanced knowledge, not average.

If Fermi and Wright could not have foreseen events for three years, imagine how hard it was to do the rest.

If you are not at the forefront of knowledge in the implementation of some achievement, and you are brought up to date on what is destined to become a leading project, you cannot predict an irreversible breakthrough based on your knowledge. Unless you can brilliantly see the perspective in some way that hunters and gatherers didn’t need, and you don’t know very well that other people may have technologies and ideas that you have no idea about. If you do not consciously compensate for your ideas, taking into account the lessons of history, then you will clearly be inclined to believe that decades have passed before this event. Three months before the construction of the first Chicago woodpileFermi no longer believed that obtaining nuclear energy was impossible or he had to wait a decade before — he was already aware of everything and saw how this could be done. But any person who was not in the know, probably thought that there were still fifty years left to this event, at that moment when the woodpile was fizzling on the squash court at the University of Chicago.

People do not automatically take into account the fact that the moment of the onset of a major breakthrough depends on advanced knowledge in this area. This range includes people who know the most and have the best ideas; so the knowledge of all other people remains average, and the average knowledge is not enough to understand when a breakthrough can occur. I think that they do not reflect on this at all, and simply make an assessment based on their own sense of the complexity of the project. And if they think somehow more responsibly, do real work on correcting their prejudices - then I have never met such reasoning anywhere.

To know that there are still dozens of years before the appearance of the ION, we need to understand the ION well enough to know which parts of the puzzle we still lack, and how difficult it is to get them; we probably will not have such knowledge until the puzzle is complete. It also means that for people outside the advanced circle, the puzzle will seem less complete than it looks in this circle.

Again, this does not mean that what people say “fifty years old” is a definite sign that something is happening on the squash court right now. They talked about "fifty years" and sixty years ago. This means that all those who believe that technological events can actually be predicted in advance by people who are not dedicated to reporting on the progress of leading projects and who do not share the best ideas about how to achieve the goal and how much effort it will take to learn the lessons. stories. Historical books contribute to erroneous ideas, carefully laying out the path of progress and all those visible signs of achievement, the importance of which we only understand now. Sometimes it is possible to predict the consequences of a big break right after it occurs, but it is rarely possible to make an accurate prediction about the time of such a breakthrough on periods longer than one to two years. And if you are one of those rare people who can predict such events, if such people exist at all, then no one knows that you need to listen to you, and not enthusiastic futurists or skeptical scholars.

Fourth. In the future, there will be other tools with which you can easily do things that are difficult today, or it will be difficult to do things that are impossible today.

How do we know that ION will appear only in a few decades? Popular articles for authorship of research laboratories usually give three main reasons:

A) The author does not know how to create IONS using modern technology. The author does not know where to start.

B) The author believes that making impressive things available to modern AI is terribly difficult, he had to spend a lot of time on the farm from the GPU, adjusting all the hyper parameters. He believes that people underestimate how difficult it is to achieve such indicators today, and panic prematurely, because he believes that anyone can just run Tensorflow and make a mobile.

C) The author spends a lot of time interacting with the AI ​​systems, and therefore is able to personally assess how stupid they are and how they lack common sense.

We have already considered some aspects of the argument A. Let us turn to the argument B.

Suppose I say: “Now a single student who has attended a computer science course can do everything in a week that N + years ago the entire research community could do with neural networks”. How big is the number N?

Several people with unknown achievements gave me the answer to this question on Twitter, but the most popular answer was five - and it seems to me correct based on my own acquaintance with machine learning. Of course, this number cannot be universal, because reality is never so simple. If you could do something in 2012, then today you can do it quite simply with modern GPUs, Tensorflow, Xavier initialization, batch normalization, truncated linear transformation, adaptive moment estimation (Adam), RMSprop, or simply stochastic gradient descent with inertia. So modern technology has become better. Of course, there are things that are inaccessible to us today using these simple methods, they require a lot of work - but in 2012 these things were impossible at all.

In machine learning, everything is arranged in such a way that if you can do something in principle, then in just a few years you can do it very easily with the help of improved future tools. From this point of view, argument B “you do not understand how difficult it is to do what we are doing” is simply illogical.

Statement B sounds similar to Rutherford’s statement in 1933, when he called the generation of energy from the splitting of atoms "a pipe dream." If you were a nuclear physicist in 1933, you had to split the atoms by hand, bombarding them with other particles, and that was tedious work. If someone started talking about getting energy from atoms, you might feel that you are underestimated, that people find your work simple.

But, of course, this sensation will always be present with engineers involved in AI on serious advanced projects. You will not be paid a lot of money for what the student can do in a week (unless you work in a bureaucratic organization that does not know what AI is; but this is not Google and not FB). Your personal experience will always say that you are paid for months of hard work. Therefore, you cannot use this experience change as a fire alarm.

People playing the role of skeptical scientists would agree that in theory our tools will be improved; but in popular articles they only write about the complexities of working with the tools of today. I think that in this mode of thinking they are not even trying to predict which tools will be available to them in five years; I have not seen such statements in the articles I read. I think that when they tell you that ION will appear several decades later, they give an estimate of how much time, from their point of view, it will take to create an ION with today's tools and knowledge. Therefore, they rest on how difficult it is to rummage in a heap of linear algebra until it spits out good answers; it seems to me that they do not at all imagine how much this pastime can change in a much smaller period, than fifty years. If they thought seriously about how their assessments of future technologies are distorted based on the current subjective sense of complexity, and tried to compensate for this distortion, I have never met such reasoning. And I have not heard that such a method of predictions ever gave good results in history.

Fifth Okay, let's be very frank. I do not think that most of the reasoning about how long to wait until IONI (or how close it is) comes from the use of modeling the future progress in machine learning. I don’t think it’s a bad model; I think the point is the absence of such models.

I was once at a conference where I spoke full of famous stars in the field of AI, and most of the luminaries nodded and agreed with each other in the sense that it was still very far from AION - with the exception of two well-known AI stars who were sitting quietly and letting others speak.

During the questionnaire, I stood up and said: “Well, you told us that progress will not be so fast. But let's be more specific. I want to know about the least impressive achievement, about which you are sure that it will not be possible to accomplish it in the next two years. ”

There was silence.

As a result, two people took the risk to give answers, and they spoke in a much more cautious tone compared to when they told us how long the ION was waiting for us. They called such tasks as “The robot takes the dishes out of the dishwasher and does not break it” and the scheme of Grapes. More precisely, it was said that "I am quite sure that according to Winograd's schemes - the results on the work with which have recently been at the level of 50-60% - in the next two years we will not get results approaching 80-90%, regardless of tools used by people. "

A few months after the conference, there was an unexpected breakthrough according to the schemes of Vinograd. The breakthrough did not reach 80%, “hurray” confidence intervals with a certain error, but it seems to me that now, when only a year is left until the end of the allotted time, would be much more nervous. I do not remember reading about this breakthrough, but they showed me a scientific work that scientists could send in just 44 days after that conference, and in which 70% of the results were shown.

But that's not the point. The bottom line is that the silence that followed my question, and that I received only two answers, very uncertainly voiced. When I asked about specific things that could not be done in the next two years, it seemed to me that the luminaries switched to the mode of building a model for future progress in machine learning, wondering what they could or could not predict that they knew or do not know. And, to their credit, most of them knew their area well enough to understand that it was actually very difficult to predict future boundaries of opportunities in a rapidly developing field, that no one knows what will appear in the scientific journal arXiv next month, and that they need to issue very wide confidence intervals with a very generous upper bound regarding

Also at the conference was Demis Hassabis [ co-founder of the company DeepMind, which created the computer system AlphaGo, winning a professional go player - approx. trans. ], so they knew that if they called something not sufficiently impossible, Demis would make DeepMind take up the task and solve it.

The style of my question was very different from talking at a conference, and required a switch of mental context: the luminaries gathered there had to ask their coarse, barely formed intuitive models of progress in machine learning, and decide what specific achievements, if any, their area development model were uniquely sweeps aside in two years. Instead of, let's say, socially acceptable verbal outpourings, designed to muffle all this fucking hype associated with ION and deserve predictable applause from the audience.

I will say straight out: I do not think that confident statements about long-term forecasts are well thought out at all. If your model has an incredible opportunity to predict what cannot be done in the next ten years, after a hundred and twenty months of workflow with arXiv, then you definitely should be able to issue weaker predictions about what cannot be done in the next two years. In your head, you should be able to build these predictions in a queue and remain in constant readiness to make them, instead of going into a nervous silence after you have been asked about them.

In fact, the problem of a two-year prediction is very complex, and the problem of a ten-year prediction is ridiculously complex. The future is generally difficult to predict, our predictive abilities in a rapidly changing and developing area are rather weak, and do not allow us to predict clear intervals at which something can not be exactly done.

Grace and her colleagues in 2017 evaluated the predictions of 352 people who spoke at ICML [International Conference on Machine Learning] and NIPS 2015 [Conference on Nervous Information Processing Systems]. In general, the speakers' predictions boiled down to the fact that the statement “all professions can be fully automated” (in the sense that “for any profession you can build a machine that will do this job better and cheaper than a person”) will not reach 50% probability in 121 years . Only here, the speakers from a random sample were asked a slightly different question: “when self-working machines can perform any task better and cheaper than human workers”, in this case they said that with a 50% probability it would be possible in 44 years.

This is what happens when you ask people to give an assessment that they cannot give, and that’s what the socially acceptable statements should be.

* * *

Speaking of the fact that for ION there is no fire alarm, I do not claim that there is no equivalent of smoke emerging from under the door.

I actually say that the smoke from under the door will always be controversial; it will not be a clear, irrefutable, absolutely accurate indicator of fire; therefore, a fire alarm that provides everyone with the knowledge that from now on your actions become rational and socially acceptable will never work.

There is such an old trail - as soon as something becomes possible, it is immediately ceased to be called AI. People working in the field of AI belong to accelerationists [ not in politicalsense - approx. trans. ] and techno-enthusiasts, they can be called members of the Kurzweil camp (I don’t consider myself to be), and often complain that these are incorrect judgments, moving goals.

This point of view does not take into account the real and important phenomenon of prejudice to the achievements of AI: if you can do something impressive looking with AI in 1974, then this is because it turned out to be possible to cheat somehow cheaply, and not because in 1974 The year's achievements in the field of AI were so amazing. We do not know how much cognitive expenditure is necessary to complete tasks, and how easy it is to cheat when performing them, so the first among the "impressive" tasks are those for which we were wrong most of all. There was a time when people believed that to win a computer from a world chess champion would require progress in the field of ION, and that such a win would be a clear sign of progress along the path of this progress. When in 1997 Deep Blue won against Kasparov, in the Bayesian sense we learned something about the progress of AI, but also learned something about the ease of chess. Considering the technologies used to create Deep Blue, we basically learned only that "it is surprisingly easy to play chess without the use of generalized technologies", and not that "we have advanced surprisingly far along the path to ION."

Was AlphaGo a smoke from under the door, a sign that the ion will appear within 10 years? Previously, people cited go as an example of what would be the "beginning of the end."

I studied the scientific work describing the architecture of AlphaGo, and it follows that, rather, the AI-technologies available to us came much closer than expected to a generalized AI, than winning the go turned out to be an amazingly simple and straightforward task. Of course, this method cannot simply be scaled to ION, but AlphaGo looks like the result of fairly generalized ideas and technologies that are adapted to the special occasion of the game of go — not the way it was with Deep Blue. I also came close to the idea that “the generalized learning possibilities of the algorithm of the human brain were not as impressive and they were not so hard to emulate usingGradient descent and the GPU carriage, as I thought, ”because if where the impressive algorithm of the brain, which has generalized learning opportunities and at the same time passed natural selection, is used, it is among people who play go.

Perhaps if we had a thousand planets Earth with a similar history, we would collect statistics and find that the computer that won the planetary championship in go is a sure sign of the emergence of ION in ten years. But I do not know. Like you. Of course, now we can also say that we just found out that it turned out to be easier to play go with the help of straightforward techniques than we thought - as it often happened in the past. There is no sign of the emergence of a real ION, no smoke from under the door, which would tell us that we really have a fire, and that ION have 10, 5 or 2 years left. Not to mention that there is no sign by which we would know for sure that everyone else will believe in it.

In any case, many leading scientists in the field of machine learning have already written a bunch of articles, where they listed their fire alarm criteria. They will believe in the inevitability of the appearance of the ION when:

A) They will personally understand how to create the ION based on the available tools. They always argue that now it is not so in order to rein in the fools, who believe that shortly before the ION.

B) When their personal work ceases to seem to them too difficult. This, from their point of view, is not understood by any plebeians, who consider that ION is approaching, because they never had to stay awake until two o'clock in the morning, trying to stabilize the generative-adversary network .

C) When they see an example of AI, which seems so clever compared to a man, that for them it will be akin to magic; but one must take into account that they know how to create certain properties of AI, and this no longer seems to them a magic; that is, they need an AI that seems smart when dealing with people; that is, they, in fact, already need a ready-made ION.

So there will be no fire alarm. Point.

There will never be such a moment before the end, when you can look around nervously and understand that you can already speak about the inevitability of ION within the framework of general knowledge about it, take measures, leave the building without panic and keeping order, so as not to be afraid to look stupid or cowardly.

* * *

As far as I can assess the situation, now that we already have AlphaGo and a couple of other warning shots, and an explosion of effort invested in machine learning, and a huge flow of scientific work, we are likely to remain in our epistemological state until the very end.

Saying that we will remain in the current state almost to the very end, I do not mean that we know about the inevitability of the IIN, or that at this time there will be no new, important breakthroughs in the field of AI. I mean that it is very difficult to guess how many more new ideas will be needed to create an ION, or how long it will take us to come up with such ideas. After the next breakthrough, we still will not know how much more breakthroughs we will need, which is why we will continue to be in such a state as before. Whatever discoveries and achievements would appear further, it will still be very difficult to guess how many more ideas we will need, and the time estimates will remain just as muddy. Perhaps the enthusiasm and funding of research will only grow, and we can talk about shortening time estimates; or we will be buried in the next “winter of AI” and it will speak about lengthening the previous estimates; but how much time we need, we will not know.

At some point, we can see a sudden influx of works from arXiv, in which very interesting, fundamental, frightening cognitive tasks will be solved with increasing speed. As a result, with the increase of this flow, even those who imagine themselves to be skeptical people get into a fuss so much that they will say that before ION there may be 15 years left, probably, most likely. And towards the end, the signs may become so obvious that people decide that it is already socially acceptable to say that the IONS may appear in 10 years. Although these signs will have to be damn clear to overcome the social barrier imposed by the luminaries, estimating the terms of the arrival of the ION through their personal knowledge and difficulties, as well as the historical unpleasant feelings of the winters of the AI, caused by the accompanying hype.

But even if it becomes socially acceptable to say that ION will appear in 15 years, in these last couple of years or months, I think that there will still be dissenters. There will be debaters who, despite solving problems with associative memory and emulation of cerebellar coordination (or something else), will still not know how to create ION. They will note that there are no AIs writing scientific works on computer science, or having a meaningful conversation with a person, and blame those who argue for alarmingly as if they know how to do this. They will explain that stupid people do not understand how many difficulties and tweaks are required for the current system to work (although modern methods will already be able to do everything that was possible in 2017, and any student will know

When all the pieces are ready and placed in places, and one of the last, available at the most advanced edge of knowledge and creativity of the whole world, is not enough, the average person will feel that ION is a terribly difficult task looming somewhere far away. because personally he still doesn’t know how to build an ION-system. The prestigious directors of AI research groups will still write articles describing the foolish excitement about destroying life on Earth and the future benefits that are available to us thanks to AI, telling us that this prospect should not distract us from real, reputable problems like credit approval systems that have learned recognize human cognitive distortions.

Of course, the future is very difficult to predict in detail. It is so complicated that I not only admit that I cannot do this, I will even make a stronger statement - no one can do that. Events can evolve according to the “breakthrough work flow on arXiv” scenario, but this is just an incredibly specific scenario that I came up with just for example. It is not based on my extensive experience of observing Earth-like civilizations developing ION. I see it as a major likelihood of the scenario “outside the Manhattan project there were no visible signals until Hiroshima happened,” since it is fairly simple. And the increasingly complex is likely to be another story with burdensome details that will be wrong.

But no matter what the details turn out to be, I predict that there will be no fire alarm that is different from the real and working ION. No preliminary error-free signals that all people will know about and agree with will allow people to act without worrying that they may be doing it prematurely. The history of technology has generally evolved quite differently even in less complex issues, such as flights and nuclear physics, not to mention more complex cases, such as ours, in which they argue about all the signs and models. We already know enough about the uncertainties and the low quality of disputes on this issue to confidently state that no socially acceptable signs of the ION approach will appear either in 10 years, or 5, or 2 years before its appearance.

It is no coincidence that no one gave any clear definitions of such a fire alarm, and did not show convincingly how much time we had left, and what specific projects we would need to start after it. If someone writes a similar sentence, then the next one will write something completely different. And none of them, most likely, will be able to convince me that they know the future and the schedule of upcoming events, or that they have determined a rational approach to solving this task, which (a) is worth doing in principle and (b) not worth starting right now.

* * *

It seems to me that the decision to postpone all matters until some vague, completely indefinite anxiety appears in an unspecified future implies a carelessness sufficient for the law of constant failures to start working.

The law of constant failure means that if your country is incompetent enough to use simple nine-digit numeric passwords for all bank accounts and loan applications, then your country is not competent enough to adjust the direction of development after such a disaster, which base of hundreds of millions of passwords came into open access. A civilization that is competent enough to change course in response to such an incentive, to respond to it the way you want them to react, is competent enough and not to make such an error at all. When a comprehensive and obvious system failure occurs instead of latent problems on the very verge of competence, the next incentive will not make this system suddenly switch to intelligent work.

The law of constant failures is especially important to keep in mind when working with large omnipotent systems or people with a high position, which you really do not want to scold. You may want to say something like: “Well, now the system is imperfect, but as soon as a suitable stimulus appears in the future, it will change dramatically, become normal and everything will be fine.” Systems that justify such hope look as if they are already doing most of the important things right, and they lack a couple of steps to conscious work. And such hope is almost never justified when a person, organization, government, or social subsystem is big.

Short-sightedness, which allows to ignore the landing of aliens after thirty years, is already large enough so that the rest of the failures do not look unexpected.

And today, when all this happens at the same time, it can be predicted that the same system with its stimuli will not give the correct results, having received an indefinite signal that perhaps aliens will land in five years. The law of constant failures asserts that if the existing authorities make mistakes, believing that it makes sense to transfer the discussion from existential risk to more real, from their point of view, problems like security of robots, then we should expect that they will talk any nonsense in the future.

People who make a large number of mistakes at the same time do not mark the wrong thoughts in their heads as “wrong”. Even with motivation, they cannot suddenly switch to the right reasoning and prediction of future steps. Yes, we conducted various experiments demonstrating that monetary incentives can reduce overconfidence and political distortions, but a) this is a decrease, not an elimination, b) it works with direct and short-term incentives, and not with unclear political statements like a lot of things, ”and c) this does not mean that the switch rotates to the full, up to and including“ conducting complex and correct reasoning ”. If someone in the brain has a switch capable of turning on complex and correct reasoning, then he has enough internal accuracy and skills to

There are no signs, signs, thresholds, after which people can suddenly wake up and start systematically doing everything correctly. People who are capable of responding so competently to any sign, not to mention the not very ideal and rather controversial evidence of the “bell” attack, have probably already calculated the entire schedule of events. They already imagined the future signs, went further, and have already pondered reasonable ideas - as Stewart Russell says, "If you know that aliens will land in thirty years, this is already a big problem."

In those days when the organization, now known as the Machine Intelligence Research Institute (MIRI), found it difficult to find funding, I learned that people who donated money last year would likely donate it this year, and people last year who planned to donate money "next year", and today they will plan to donate them "next year". Of course, there were transitions from zero to one; everything that happens must happen for the first time. There were students from colleges who said “later” and did not give anything for a long time, and then they got a good job and started to make donations. But I also learned that, like so many cheap and simple consolations, the word “later” produces addiction, and that this luxury is inherent in both the rich and the poor.

I do not think that with the issue of regulation of IONS something will be different. People who try to deal with the problem of regulation as far as possible will have a little (or a lot) progress in solving the problem next year than in the past (of course, given any breakthroughs that have occurred during this time). People who want to postpone the problem later, when there is a better understanding of AI and IION, after progress in this area next year, they will want to postpone this problem for the future, when an even better understanding of AI and IION appears.

Some people want regulation to be, and therefore they are trying to figure out how to make a reinforcement learning system reliably determine the utility function over certain elements of the causal environment model. Others like to work on other things more, so they prefer to declare that today there is no work to be done, and not to think in advance for a couple of hours about what can really be done. And this will not change tomorrow, unless we see some interesting news headlines tomorrow, and even then it most likely will not change. The luxury of saying “later” is not only available in the absence of good options.

After some time, I started talking to supporters of effective altruismin college: “If you want to earn more to give more, for now donate $ 5 every three months. But never donate the same amount or the same organization twice in a row to practice the habit of revising goals and revising the amount of donations. Do not instill in yourself the habit of always saying "later".

Similarly, if someone is going to work on regulating ION later, I’ll tell him to spend a couple of hours every six months on building the best current scheme for regulating IION and on useful actions in this direction. Assuming, if necessary, what IION can do with the help of technologies resembling modern ones. Publishing your current, not good enough scheme, at least in Facebook. And let them be ashamed of a scheme that does not look as if someone really thought over it for two hours trying to find the best approach.

Something related to AI will be better understood in the future, and something will help us gain confidence that certain research methods are directly related to the creation of an ION. In the future, sociological events may occur, similar to the publication of Nick Bostrom's book “Superintelligence,” Ilona Mask tweet about this, running a stone in Overton's window, or joining this event of such a more respected star as Stuart Russell. In the future there will be more events similar to the appearance of AlphaGo, which will be able to publicly highlight new breakthroughs in machine learning. It may happen that this does not leave us in the same epistemological state as we are already after seeing AlphaGo and the generative-contention network. This can happen! I do not see how, but the future has the opportunity to organize such surprises.

But before you start to expect such a surprise, you need to ask yourself whether the uncertainty of the appearance of the ION is so uncertain. If it seems to you that assumptions about the 50% probability of the occurrence of an ION in N years are not enough to act, ask yourself how you would feel if you believed that the probability of the occurrence of this event in N years equals 50%, and all the rest the inhabitants of the earth also believed in 50% in N years, and everyone believed that it was permissible and the policy P should be pursued when the probability of the occurrence of an ION in N years is 50%. If such a thought causes you to have other sensations, then any uncertainty related to whether P should be done is not related to whether we have to wait more than N years until ION appears.

And almost certainly you will still experience this uncertainty, no matter how strongly the appearance of the ION approached. No matter how long the ION appears, any signs will almost certainly not lead to the generally accepted public knowledge that the IION has a 50% chance of appearing in N years, therefore there will be no agreement that it is acceptable to react to it, pursuing a policy of P.

And if all this becomes public knowledge, then P is unlikely to be ignored by the intervention, and regulation of AI will be ignored by the problem; by the time you have been waiting too long, and it will be too late.

It is more likely that there will be no public knowledge, therefore there will always be some uncertainty about the necessary actions.

You can act in spite of this, and you can not act. At best, you will not act until it is too late to change something seriously. On average, you will not act at all until all of this actually happens.

I do not think it is very clever to expect an unnamed epistemic miracle to change our sensations. You will most likely be in such a state for quite a long time - including this nervous uncertainty. If you deal with this condition by saying "later", then any procedure of action is unlikely to lead to any good results for planet Earth.