Why do plants need machine learning

    We learned from Roman Chebotarev ( convex ) how machine learning is being implemented in industrial enterprises, who have achieved the greatest success in this and what examples of use already exist . Roman is an ML architect and director of implementation at Tsifra . For 11 years he has been implementing smart technologies of the class Machine Learning and Artificial Intelligence. For the past few years, Roman has specialized in ML / AI in the industry.


    Tell me about your professional

    path. I began my professional path with machine learning (although this term has not yet been widely used) for computer vision problems. I developed various modules for video analytics systems: crowded detectors, smoke detectors, object counters. Then they were planned as security systems of the future generation - now they are used everywhere.

    Slowly from image analysis, I switched to data analysis in general. I already worked at CROC, where I came as a developer, and left as the head of machine learning practice. B ofMost of the experience I got there, and basically we solved the problems associated with the prediction of various quantities in the future. More tasks were in retail - machine learning then was most in demand among customers in this particular area. We solved the problem of demand forecasting to optimize logistics. There were a lot of such tasks in different areas: from food retail to automobile gas stations.

    Then a serious interest in machine learning began to emerge from industrial enterprises. At some point, my partners and I decided to organize my own startup, Theta Data Solution. We made 6 projects and more than 10 pilots per year for industrial enterprises, and then our company was acquired by the company “Tsifra”, where I now work as director of implementation in the department of AI. Compared to the original startup team, we have greatly expanded: now there are more than 30 people in our AI-tribe (as we call ourselves).

    When did industry become interested in machine learning?

    Interest has always existed, but the willingness of companies to invest in these projects, albeit very, very sluggish, appeared, according to my observations, in 2013. It became more or less possible to call it a trend by 2016. Now there is a phase of rapid growth.

    What are the specifics of designing machine learning models for industrial enterprises?

    The industry has a very high cost of error. If you start managing some kind of installation incorrectly, at best it will start working poorly, not as efficiently as it could (in any case, these are very large numbers in rubles), and at worst irreversible processes will occur and expensive repairs will be required.

    This affects how models are designed and operated: they are strongly “skewed” in the direction of less experimenting. If, for example, in FINTECH or telecom, you can afford to do A / B tests at a subset of clients and show them fundamentally different advertising, give a directional discount, etc., then the industry has a lot of tools and experimentation . If you try to formulate the features of one phrase - drastic changes in the work of the industrial system can not be done. The changes will be very small and directional. We will make adjustments, watch what happens - and, if everything goes well, try to move in this direction, but in very small steps. This is called regularization of machine learning models (more precisely, management models) - in the industry they are very regimented for change. This minimizes the possibility of costly mistakes.

    The second feature is that machine learning models (ML) need to be friends with physics and chemistry. It turns out to be quite difficult. A model built on data doesn't care what kind of physics lies behind a particular process — it just picks up the patterns between the data. It often turns out that the model constructed is completely nonphysical. For example, any sane person understands - and physics confirms this: if the heating system temperature is raised, the temperature of the heat carrier in the houses will become warmer with other things being equal. A model can learn a completely different relationship, sometimes the opposite. To give more weight to indirect factors (temperature fluctuations in the street, time of day, etc.) and due to this, it would seem to learn the correct behavior, but using the “wrong” factors.

    Now people working in the industry are quite open to new technologies. They are trying to figure out what we offer them to deliver. If the models are non-physical (as checked by a couple of simple tests), then no one will give the green light to launch such a system. But in the end it turned out that as a result of such failures, we found another, by current estimates, a much more efficient way.
    There are laws, theoretical or empirical, systems of differential equations and a huge body of knowledge that were created by physicists and chemists. This knowledge is used in the design of installations and, in general, describes the production process more or less well. We incorporate this knowledge together with ML to obtain physical models - in fact we rely on a set of well-known dependencies and diffuros, specify the coefficients using available data, and also use standard ML approaches (boosting) to describe the dynamics that could not be “learned” by physical approaches. .
    For clarity, I often introduce such a thing as “spend data”. When you teach a model something, you “spend” the data (in the sense that any reuse during training is a delicate point, there is a risk of overfitting). So, we are not “wasting” the data to restore patterns and dependencies, which in general terms are already known thanks to scientists and technologists. We use these well-known dependencies and “spend” data on refining the characteristics, completing the unrecorded dependencies in physical models, and ultimately building models that take into account the characteristics of each local production site or even equipment, knowing how it works in principle.

    As a result, we get better and more stable models. Naturally, physical and chemical models of processes are not always available or complete - in this case, we have analysts with experience in relevant industries who could build appropriate baseline models for data scientists.

    In addition, we try to use the approaches of the theory of automatic control to make decisions about optimal control parameters that need to be set on the installation, taking into account the inevitable lag in time and the likelihood that the recommendation will not be accepted at all. In general, we look at the Reinforcement Learning approaches, but so far the resulting control laws (policy) are quite unstable in our tasks. But behind the unification of these approaches lies exactly the future. And this is not just my opinion.

    With such a “physical” approach, an important long-term consequence has emerged over time: due to the greater stability of such models, we wake up on a call less and less at night that something went wrong and the model should be retrained. As a result, we spend less time on support.

    Many people in the world came up with such a hybrid approach, but in Russia we were among the first to go beyond experiments and put it into real production.
    On November 22, Roman will be the moderator of the discussion panel “AI and IoT: Waiting and Reality” at AI Conference. Details and program of the event - on the official website .
    How is the work on creating a digital model of the production process?

    The project itself for the development and implementation of a little different from other industries. In general, project managers who come, say, from the banking industry to the industry, feel quite comfortable (except that technologists are usually teasing them). From an organizational point of view, projects are not very different. First, we fix the customer's expectations - what they want to achieve. Sometimes we offer them to work together if they do not know what they want, but they really want to digitize. Together we find improvement points, put them into some measurable KPIs, do prototyping, do a little research or even a pilot - we convince ourselves and the customer that these KPIs are achievable, then we develop models, use a large number of our current developments,

    Key features focus on the implementation phase. Systems are quite complex, both in how they work and in what data they use to make decisions at different points in time. Workers at the plant often do not have specialized education to work with them. Therefore for them it is necessary to invent special deshbords and mnemonic schemes, to conduct trainings. At the same time, there is a guide that is quite well versed in what they need, and for them you need to do other dashboards, with more detailed information.

    In general, the main "enemy" of our systems is a process engineer. Decisions on regime change are made by him, and he usually has his own opinion on how the shop or production site entrusted to him should work. It takes a lot of time to convince direct implementers to trust the recommendations of the system. More precisely, not just “to believe”, but to take and test it - at first just look at the recommendations, then apply it to the point. Often these employees are not directly subordinate to the direct customers of the project and it is not possible to force them to follow the recommendations in the directive form. But we in general seem to have learned to build such dialogues and processes of persuasion at different levels, from impenetrable operators to harsh production managers. This is a very interesting experience. especially for such “vanilla” mathematics mathematicians from Moscow like us. But, as is usually the case, the real deal is better than any persuasion, so if our models really work, then this is the best argument and usually such discussions are short-lived.

    How often in the development of the model and its implementation do you have to go to a real enterprise?

    Business analysts spend the most time on the site. They are always present in the project team, in addition to data scientists and data engineers. Business analysts describe the processes, write the rules and restrictions of the system, and they need to deeply understand the process that is to be, as it is now fashionable to say “digitize”, or rather, forgive, “digitize”. At the site, they find out certain nuances and understand where, how and what needs to be implemented in order for the process to work: how they usually manage the process, how they don't manage, what they usually don't write in the regulations. A lot of things can be found out only in the smoking room, after talking with local hard workers during the break - how things really are, where you really should make an effort, etc. The task of analysts is to reveal the need, But this can only be learned from real employees who work in the field with their own hands. But there is a specificity: those people who work with their own hands usually live far away from million-plus cities. Sometimes they are generally present on a rotational basis at the fields and quarries. Therefore, we have to go to them in different scenic spots.

    The most distant, where did you go?

    We were everywhere, from the Murmansk region to the Khabarovsk Territory.

    Does it often happen that the created virtual model starts working immediately and without surprises in real conditions?

    We try to minimize all the surprises at the survey stage, but when implemented it is never without them. Surprises can be divided into several groups. The first one is, of course, IT and infrastructure. To update models in time, it is important for us to have access to the data in order to change something, fix, add. But access to the infrastructure may not be, if the object is located somewhere very far away, where the connection is organized, as we say, “through a comb” or not at all. If this is known in advance, you can build and debug a process that will update the model independently, without the intervention of its creators. This is being done relatively easily now, we have ready-made technologies for this - but nevertheless, I would like to know in advance that there will be no connection. At least, because it affects the labor costs and the cost of the project. Project customers most often go to negotiate with IT people when the project is already close to implementation. This is characteristic not only of industry, but here it is most critical. On whether the Internet will be or not, the solution architecture strongly depends, as I said earlier. And it's not just models.

    The second class of problems is associated with incorrect data entry. For example, data on the quality of certified products, laboratory data. This can happen for various reasons, I will not talk about them, most of the reasons are not very pleasant to voice, and even less to hear - but this is a very big problem, because a model learned from inaccurate data begins to predict the inaccurate characteristics of the process and issue incorrect recommendations . This can wipe out the entire project.

    Remember the most successful and most time-consuming example of implementation.

    I will start with a successful project in power engineering. We saw the customer only two times. For the first time we arrived we clarified the task, provided us with the information we needed, we left and called each other once a week. Three months later, the first release was rolled out, after two months - the final release. Everything worked perfectly, the models are updated automatically and the system has been working without failures for more than two years. The project required a minimum amount of effort, because the customer was very competent: he understood what he needed, how it should be managed, and we knew about all the nuances beforehand.

    Labor-intensive examples are much more. Unfortunately, the presence of the term “digitalization” in preliminary conversations with the customer here is often a sign that the project will not be successful. Often we hear: "You are involved in our digital transformation process, we are completely reworking everything, so screw your AI here." At the same time, people often do not understand that they should solve problems not with the help of a machine, but first by changing the processes in their company to more appropriate “digitalization”. Changing processes (or at least rethinking them) should always be the first phase of change in any digitalization or other evolution. Any tool, including machine learning, has limits of applicability. If the process is ancient, non-optimal, and even worse - built entirely on the consensus of people (several people need to sit down and decide what to do - this often happens in production logistics, where production workers, logisticians and commerce encounter), then no machine learning will fix this. And, on the contrary, sometimes the simplest changes in the processes (for example, the concept of “lean production”) make it possible to achieve such effects that no ML can achieve. Unfortunately, very few “transformers” understand this and work in this direction. Hiping on the introduction of AI, no matter why, is more common practice. sometimes the simplest changes in processes (for example, the concept of “lean manufacturing”) allow to achieve such effects that no ML can achieve. Unfortunately, very few “transformers” understand this and work in this direction. Hiping on the introduction of AI, no matter why, is more common practice. sometimes the simplest changes in processes (for example, the concept of “lean manufacturing”) allow to achieve such effects that no ML can achieve. Unfortunately, very few “transformers” understand this and work in this direction. Hiping on the introduction of AI, no matter why, is more common practice.

    A simple example: there is a distillation column, in it you can control the rates of steam and reflux. If we just issue recommendations to the operator on the screen - “buddy, twist this pen like this” - then the effect of the system, unfortunately, almost will not. Man ideally should remain only for control, and direct control should be automatic. Such a change in the process according to our very conservative estimates gives an improvement of 3-4 times. I am not in favor of dismissing and replacing all people with cars - just a small change in the process with very little investment gives a much greater effect.
    Many projects, about which it is claimed that AI has been implemented there, actually look like this, forgive the uterine truth: some uncle Vasya has recommendations on the screen, he looks at them and says, “Yes, and hell, maybe tomorrow I will put it the way he wants - and today I will not do anything. ” It happens very sadly that powerful cool technologies are broken about the processes of an enterprise and people who are not ready to change these processes. But if this uncle Vasya put the KPI to implement the recommendations of the system. Or even without AI at all - to put Vasya KPI on the specific output of the product to the raw materials, just as a bonus to salary - then there are really serious effects. Provided, of course, that Uncle Vasya cannot be replaced with a controller, but this is a question from another plane.
    How are businesses doing data collection and machine learning? How many of them are trying to go in this direction?

    Statistics on the number of enterprises is improving every year. The leaders, as usual, are those who have money and the opportunity to invest in long-term effects: oil industry, petrochemicals and metallurgy. Everyone else is catching up.

    But you need to understand that basically these are systems that give recommendations to a person, and he already decides whether to do something in accordance with these recommendations or not, there is practically no automatic implementation of recommendations. This is definitely a stop for the development of these systems. In general, this, of course, has never been Industry 4.0, as it is often liked to position in the media. But the re-equipment of automation requires large capital expenditures, so for now we rejoice in what we have.

    We would like to see processes in companies more organic: people first collect data, and then implement machine learning based on them. In fact, at first there is a need to do something based on AI / ML, we come to the customer and understand that the necessary data is not collected. Or they add up in a terrible way, so that they cannot be reached - you need to start a data collection project. About 5-7 years ago, it was common in telecoms and banks everywhere (now it is not) - today the industry has the same problems. There were projects that were delayed for six months and a half years due to the lack of data.

    Is this the time during which sensors and data acquisition systems are being deployed?

    Almost everyone has sensors - the question is that the data from them can either not be stored or stored in some kind of short-term storage for three months, for example, so that it can be arranged on the basis of their analysis of flights. As unnecessary, they can no longer be stored, and if they are stored, they are not suitable for analysis. We have to do the processes of their extraction and purification. And there are altogether comical cases when everything seems to be there, and we arrive at the enterprise - and there everything is warm analog lamp , for example, dial gauges.
    We believe that any process that is not yet automated with AI and ML can be optimized by 1-2%. When we choose a project, we analyze: how much in this industry, in this workshop, money is spent on raw materials, on electricity, on repairs? We take this value in money and calculate 1-2% from it. If in monetary terms, this is reasonable, then we are engaged in this project.

    Often we work according to the scheme of success fee. Conventionally - the introduction of the project costs 50 million - various market integrators, for example, call you about such a cost. We are ready to make a project for 10 million, but at the same time we want to receive a percentage of the savings in the next 2-3 years. As a result, due to this, the amount can go in our direction and 70-80 million or even more - provided that the customer also earns. We are conducting pilot phases, we can see what effects we can achieve, and are ready to work according to such a scheme - to receive a payment that is proportional to the effect achieved.
    What standard types of tasks does AI solve in production?

    The most common task is to predict the failure of equipment, more precisely, to diagnose moments of atypical behavior. There are some peculiarities: we need data that may not be collected, we need information on how this equipment works - for this we have production personnel with whom we consult. Because some patterns in the data are logical and do not mean that the equipment does not work correctly.

    An example of such a task is to determine how long a section of a pipeline can work, depending on where it is buried, how deep it is, what is shown by the latest data from an internal survey of pipes or magnetic control, how often the modes change and how they were. We can predict when the pipe will become unusable and optimally plan its replacement.

    The second type of tasks is associated with the need to optimize a process. Let us consider the example of heat and power, as the most understandable to the general reader. We can control thermal conditions at heat sources (boilers, CHP, etc.), while we must maintain a certain temperature level in different rooms: they are at different distances, built of different materials, differ in geodesy and as a result differently cooled by ambient air. How best to build thermal conditions in the boiler room or CHP, to withstand the quality level indicator in relation to the end customer? Here it is necessary to determine the main indicator of efficiency. We can spend totally less energy for heating and transferring the coolant, we can reduce the number of complaints from frozen grandmothers, we can reduce variable heating costs, reduce heat loss or even equipment wear. You can do any optimization model - just tell the relative priorities of various factors. This choice is the biggest problem. Imagine yourself as the owner of a thermal power company. How many dissatisfied grandmothers are you willing to exchange for the fact that this pipe will live for a couple of months more? Extremely difficult question. Therefore, our business analysts are working, among other things, on helping to reduce all factors to the ruble as the most universal measurement value. After that, it usually becomes clear what you need to work on and what to optimize. This choice is the biggest problem. Imagine yourself as the owner of a thermal power company. How many dissatisfied grandmothers are you willing to exchange for the fact that this pipe will live for a couple of months more? Extremely difficult question. Therefore, our business analysts are working, among other things, on helping to reduce all factors to the ruble as the most universal measurement value. After that, it usually becomes clear what you need to work on and what to optimize. This choice is the biggest problem. Imagine yourself as the owner of a thermal power company. How many dissatisfied grandmothers are you willing to exchange for the fact that this pipe will live for a couple of months more? Extremely difficult question. Therefore, our business analysts are working, among other things, on helping to reduce all factors to the ruble as the most universal measurement value. After that, it usually becomes clear what you need to work on and what to optimize.

    What types of tasks it became possible to solve quite recently due to the improvement of the MO methods?

    I, probably, will disappoint the majority of readers, because the movement is not due to the use of recent advances in ML methods. Not because what takes root in production should be time tested and more sustainable. Here the development goes the other way: the model needs to be friends with physics and chemistry, which I have already mentioned before. It turns out that this is also very difficult from the point of view of ML.

    Give examples from your practice, when the decisions made by the machines were more successful and more efficient than those that came from a person.

    In fact, the decisions and recommendations that are issued by the system are always ultimately more effective than those issued by man. Otherwise, there would be no point in our business. Here are some examples.

    In steel production, blast furnaces consume energy like a small town. Depending on what quality of scrap we fall asleep there, what size its pieces, you can adjust the strength of the current that is supplied to heat the furnace. By controlling the current strength, it is possible to significantly (and for industry 1-2% - this is significant) to reduce the cost of electricity.

    Another of metallurgy - furnace-ladles, in which steel is brought. When melting, ferroalloys are added to steel. They cost specifically much more expensive than the main raw material. Analyzing the characteristics of a particular material, we understand when you can pour a little less ferroalloys to obtain the desired quality of the product and at the same time save on ferroalloys.

    In the oil industry - we have optimized the mode of operation of the pumps during the mechanical lifting of oil. We have learned to slightly increase the flow rate of extracted oil simply due to more efficient control of pump modes. It is important that at the same time we minimally use geological data due to the fact that our management horizon is not very long (up to a month) and we manage to avoid integration with very complex and expensive formation modeling software.

    All production in Russia are rare, and to say that we are working somewhere means immediately opening the customer and breaking the NDA. Therefore, let's say that we are able to do the same things to optimize the production of mineral fertilizers, various chemical plants (not from petrochemistry). From the open - the Digital Plant project for PJSC Gazprom Neft, the details of which are easy to google.

    About our blog
    Tomorrow we will suspend our activity on Habré, so this is our last post for the near future. But the announced AI Conference and those events that we have not yet told , remain valid. Thanks to everyone who read us.

    Also popular now: