We answer readers' questions: what is the IBM Watson cognitive system, and how does it work?

Alexander Dmitriev

Good afternoon, Habrahabr! Today, Alexander Dmitriev, business consultant at the IBM Client Center in Moscow, will tell you what the Watson cognitive system is and how it works. He will answer questions that readers have had while reading other materials on this topic.

Alexander, our readers regularly ask questions at Habré, the main message of which can be put into one: “What is the IBM Watson cognitive system, and how does it work?” Please help me answer it.

Hello. First of all, Watson as a whole is a large set of software packages that use a wide variety of algorithms. Some of these packages are available in the cloud, and some are for local deployment. IBM has assembled a variety of analytical modules and built a system that can handle a truly huge amount of data. This system works with both digital and textual information in various languages, including Russian - still at a basic, but still quite deep level.

When processing information, relationships and correlations are established between a wide variety of data, events, facts and phenomena. One of the main tasks of the system is to identify relationships that are invisible to the simple eye and which cannot be detected in the usual way, or it is difficult to do this using standard methods. For example, if a company has a monitoring system that provides thousands of changing parameters per minute, then no analyst is able to analyze such information quickly. The Watson toolkit allows you to do this. This is if we talk about IBM Watson in general.



I have seen questions regarding whether Watson is artificial intelligence. The answer is no, this is not artificial intelligence. This is a kind of amplifier of a person’s natural intelligence, which allows you to process information faster, cover large amounts of data and find what passes by the human eye.

Stanislav Lem wrote about this in his book “The Sum of Technology” : “Man cannot directly compete with Nature: it is too complex for him to stand alone against. Figuratively speaking, a person must build between himself and Nature a whole chain of links in which each subsequent link will be more powerful than the previous one as an amplifier of the Mind. ”

How is this done and why? There is a Watson-based top-level analytics system and lower-level analytics systems. The latter are actually search and analytical systems with certain specifics. They solve applied problems. How it works? Pour a large amount of information on a certain topic in the form of files of common formats such as xls and csv. We upload this data to the cloud, after which the Watson Analytics system starts analyzing this information, finding correlations on its own - with minimal operator involvement. This is a small, but very important difference from other systems, because here it is not just a search for previously downloaded data. I emphasize - the system itself analyzes the downloaded information.

What does it mean - herself? The system is so configured that it looks through all the downloaded data, cleans it up, indicating technical problems like format mismatch, gaps, and omissions. A person throws out all extrema that are errors or an occasion for a separate consideration, chooses a processing method. Then the system analyzes the data, searches for correlations, finds the strongest ones and shows the operator several hypotheses with correlations, say, from 0.3 to 0.8.

I would call these tasks conditionally lower-level tasks. They are designed to simplify and speed up the work of the analyst. Routine operations are automated by the system itself. This is if we talk about Watson as a system for finding correlations in big data arrays. What six analysts need in about a week, the IBM Watson Analytics system through the cloud does in about two hours. How hard is it to work with Watson? I once conducted an experiment, sitting down for a system of people who are more or less versed in statistics. They saw the interface for the first time. After an hour and a half, they already independently and very actively worked with her.

The upper level is large systems, the implementation of which requires considerable time (from six months or more). Their working principle is based on the idea of ​​a Gartner study, which states that by 2030 the overall qualifications of specialists in most industrial sectors will decrease significantly. One of the factors of this can be explained. The fact is that a specialist who is used to (and who needs to do his duty) constantly to use reference information no longer considers it necessary to remember everything that experts remembered from the “old school” (from the height of the stratosphere to the boiling point of copper). The new generation willingly resorts to the Internet as a reference and does not keep all the necessary knowledge in mind. It turns out that the specialist becomes dependent on machine systems to a certain extent, and the general level of his qualifications decreases accordingly. This time.



Second, why do we need such complex systems? Oil producers and many other corporations have a huge problem - the transmission of information from “generation to generation”. For example, the previous team of employees has accumulated an archive of very valuable technical information. But here's the problem - no one is able to read it. After all, this takes a huge amount of time. In order for a specialist to get acquainted with this information, it will take several years - this person should read for days on end without food or rest.

So, training new employees is a costly problem. Change of personnel at large enterprises - there may be hundreds and thousands of specialists per year. A man left - and invaluable experience and knowledge left with him. How to transfer experience? By recording? We talked about them above.

It turns out that at the level of transnational corporations, where a huge amount of data, nomenclature, hundreds of thousands of people, it is necessary to create a certain system that would accumulate data of a certain thematic specialization. Ideally, this system can be used not just as a reference, but as a reference that provides advice.

What is the purpose of the Watson top-level system?

A huge variety of analytical packages included in the general Watson toolkit are put together, from them the necessary packages are selected that will process information using a specific method. Well, after that, any type of data is loaded into the system - digital, meeting minutes, business correspondence and negotiations, communications, contacts, prices, equipment nomenclature, textbooks on the oil industry, reports for various periods of time. This may take more than a year, but as a result, a pool of the corporation’s main knowledge base is created, which can be actively used.

After that, algorithms are set up that allow you to analyze information, detail it, highlight and build a tree of relevant topics - for the same "oil industry", this is equipment setup, reservoir development, statistics, technology trends, etc. All this is collected by topic, a hint system is being created. Systems of this kind, developed by IBM specialists, are already operating in a number of corporations, including the Australian company Woodside Petroleum.



The foregoing can be illustrated by example. There is a chief engineer at the enterprise, he gives the task to drill a well in the reservoir, for which there is relevant data. The one who was given the task turns to the system in natural language: "What needs to be done to drill a well in such a layer to such and such a depth?" And the system gives an answer, it works as a hint for a specific oil task. A system configured for the "oil industry" makes a selection of documentation with conclusions, and "says" - they used to do this before, but at the same time there were some problems that could be solved like this. This is the Watson system - it suggests what a person needs to do in a particular case, acts as an assistant.

Can the Watson system work as an oncologist, meteorologist, someone else?

As an adviser or assistant - yes, of course. IBM Watson refers to a common product system for any application. But in each case, it is necessary to configure the system to solve specific issues.

In the case of oncology, this is the creation of a database for a specific disease, for example, lung cancer. A huge amount of data is loaded into the system, including depersonalized patient records. After that, the doctor asks a question about the method of treatment for a particular patient, and the system gives an answer based on the individual characteristics of the person. Watson does not assume the functions of a doctor - anyway, it is the doctor who will diagnose and prescribe the treatment, but it helps to personalize the treatment, clear the necessary data from errors and make a selection of the best treatment options for this particular patient at the moment.

It is important that the system also checks all data for legitimacy and errors, as there may be errors in the same medical data. The problem of doctors (and not only doctors, but modern specialists in general) is that they do not have time to learn anything new. This is not their fault. Simply, if there is a lot of work, and a qualified specialist always has it, then there is not enough time for training. Therefore, the same doctors often use not the most modern methods. And the Watson system can offer a new method of treatment, even several methods, with a certain probability of curing the patient and a fixed degree of risk to his health. And the doctor, in consultation with the patient or relatives of the patient, can make a decision based on these data. Once again, it’s worth emphasizing that the responsibility lies with the doctor, After all, the answers of the system are advisory in nature. The system helps the doctor by providing the latest information on which methods are suitable for a particular patient.



How does IBM Watson work with natural language? Can a system understand the context of a literary work?

Definitely yes. But the question is why? Another question - who needs it, and who pays for it? When working with the language in terms of processing a literary work, it is necessary to consider the text in connection with the historical context of the work itself. The system can understand everything if it is posed such a task, including the works of O. Henry, whose translations were best obtained from Korney Chukovsky. I must say that systems that work with the language are also configured and trained. In the simplest case, this is trivial parsing, that is, parsing clearing the text of redundant information. As for Watson, this is, first of all, the creation of dictionaries of different languages. In any case, the system must be trained with an eye to a specific task.

I personally participated in a project of emotional analysis. Today, Watson captures the emotional coloring of the text. For example, she learned to define irony. In general, here again we are talking about revealing correlations. As for the same irony - it was invented by the ancient Greeks. It seems that any person recognizes her by some specific signs. If a machine is taught to capture these signs, it will also learn to detect irony.

I repeat, the capabilities of the system are determined by the relevance of the problem being solved. Basically, large companies need help with IBM Watson, which hardly needs to first determine the irony in the reports of their employees (although this probably happens). But for them, if necessary, we configure the system in such a way that it can determine the attitude of users / customers to brands and products of companies.



Example: in Spain, more than two years ago, a large-scale project was carried out to assess the attitude of users to the brand. The customer was a large company, which asked to analyze the attitude towards it according to various sources, including social networks, newspapers, magazines, etc. This was successfully done. In the course of such work, we isolate and analyze false data that are related to fakes, which cast a shadow on the reputation of the original brand. At present, world-famous brands are using this system; the project is very successful and allows increasing sales efficiency.

In general, Watson solves specific problems. The system can do a lot, namely, it is determined depending on the general statement of the problem and its orientation.

Question - the limits of the system. Let's take an example - if we take the same O. Henry, is it possible to set Watson to a literary translation of the works of this author, and how long will it take? Say this was needed by a publisher that was willing to pay for it.

The answer is definitely possible. But I can’t answer exactly how long it will take. This is a matter of effort and investment.

Any specialized Watson system, whether it is a doctor, financial analyst or engineer, requires the participation of specialists. In this case, I would recruit teams of the best linguists in the theory and practice of language. Part of the teams will be dictionaries, idioms, search for text data, correlations between Russian words and English. What for? One word in any language can mean a lot. Such words will enter the dictionaries, indicating the widest possible range of their knowledge.

After that, you need to start solving the second problem. Namely - to drive into the database texts of translations by O. Henry, which are considered the most high-quality and successful. Then Watson will use a technique for assessing correlations with maximum values ​​to search for words that are meaningful. The system will choose various translation options from simple to complex (words, phrases, sentences, etc.). During this process, expert groups will be needed to further train the system. They will adjust the translations, fine-tune them so that after such training the Watson system starts to produce really good translations. That’s how it works - the first translation will not be very good, the second will be better, and then very good. A big plus of Watson is that the system can be tuned, thanks to the feedback. Indeed, without feedback, the system will simply lose control. Feedback allows the system in dynamics, in the course of work to refine and adjust the main goal. In our case, feedback is provided by specialists in the subject area. If it is an oil company, like Woodside, then the best experts will note the best, successful answers of the system, and the system will remember this, gradually improving the quality of recommendations issued. So Watson has another advantage. If most systems become obsolete over time and require rework, then this system only gains experience over time and becomes even more powerful. If it is an oil company, like Woodside, then the best experts will note the best, successful answers of the system, and the system will remember this, gradually improving the quality of recommendations issued. So Watson has another advantage. If most systems become obsolete over time and require rework, then this system only gains experience over time and becomes even more powerful. If it is an oil company, like Woodside, then the best experts will note the best, successful answers of the system, and the system will remember this, gradually improving the quality of recommendations issued. So Watson has another advantage. If most systems become obsolete over time and require rework, then this system only gains experience over time and becomes even more powerful.

Another question - are there any problems that Watson cannot solve now under any conditions?

There is a very important aspect - ethical. Part of the tasks is unsolvable because existing issues go beyond the scope of technical systems. An example is a robomobile. Roughly speaking - on whom or what will the car hit, if it is impossible to avoid a collision, but there is a choice - say, against a wall, an elderly person or a pregnant woman? A human driver will make his choice anyway. But the car - no, it cannot make a choice, since this issue has not yet been resolved either legally or ethically. And in the car, this knowledge and rules of behavior in extreme situations simply can not yet be laid. This is the first class of problems that are not yet solved, since a number of ethical, legal, social and other problems associated with the tasks themselves have not been solved.



The second class is extremely complex technical tasks that require a huge amount of resources. In order to understand whether there is a solution to such a problem, you need to at least try to solve it. An example is the same as with the texts of O. Henry. Nobody has done this yet. Probably, if you try, then everything will work out, but for sure we can’t say right now.

To summarize the above, I want to express confidence that almost any problem can be solved. If now it seems that it is impossible to get an answer to a question, after a while there is a person who gives an idea that opens up unprecedented opportunities. Example: at one time it was believed that the composition of stars could not be determined, and this could never be done. But the spectrograph was soon invented, and it was immediately determined that the prevailing element in the composition of the star was helium. After some time, the composition of the stars learned to determine very accurately.

Regularly there are solutions that radically change the vision of our world. The boundaries of the possible are difficult to establish and, frankly, I would not even set them.

How do you see IBM Watson in the future?

As mentioned above, Watson in general can be described as a system that helps a person make a decision in conditions of great uncertainty. I believe that, like all other systems, it will become significantly cheaper, more universal, it can be used in other areas.

I think that this will be a universal hint system that can answer a wide range of questions and which will become familiar to us. Moreover, she will answer not like modern search engines - not just give links to Internet sources, but provide recommendations indicating the source of information and the methods used.