Three Microservice Stories, or MSA for Enterprise
The idea of a microservice is to build an application as a set of small services with dedicated functionality, each of which works in its own process. This approach has several advantages, but this is not the topic of today's story, but how the idea of microservice architecture looks from the point of view of Russian corporate business and IT managers in enterprises.
Together with Igor Bespalchuk we will try to look at this trend from three different angles, which is very useful for understanding the nature of what we are dealing with, and, as a result, in order to draw the right conclusions and make the right decision.
Microservices are one of the most important and significant components of the Web-scale architecture, which has the greatest consequences for redesigning the device of techniques and patterns in Enterprise. It is difficult now to say in which area the technology itself is now located - maybe at the highest peak, and we still have to give up ten more times. But, nevertheless, this is not a reason not to study it right now .
About the speaker: Igor Bespalchuk - application architect at the CUSTIS group of companies, which develops custom software for large corporate parties and government agencies. Prior to this, Igor also worked with corporate companies and retail, so he had the opportunity to observe how development processes are taking place in the corporate sector and how the management system works, as well as how certain changes are gradually leaking there.
Next is the transcript of Igor's report at the Web-Scale IT Conference 2017.
I first came across microservice architecture in November 2012 when I came across a wonderful presentation by James Lewis with QCon "Services: Java, the Unix Way" . He talked about how they worked on a large, complex, loaded project with a huge amount of data: they sawed it into separate functional elements, and decided to implement them as separate services. It was so amazing and unusual, including for our practice of architectural design, that directly excite the imagination. And at the same time, this intersected strongly with the concepts of Domain Driven Design (DDD), which we used quite deeply in our practice for many years, but in monolithic systems.
And in 2014, on the website of Martin Fowlera large article “Microservices” appeared , where he clarified what microservices are and why they are needed, the article is translated into Russian .
In a nutshell, we used to do heterogeneous functionality together, there was one system or application, and it was executed as a single process. If it was necessary to scale the system, this was done by horizontal scaling, but uniformly throughout the functionality.
In short, the central idea of microservices is that we begin to put a separate type of functionality in a separate process - Bounded Contextas it is called in DDD. If necessary, we can scale these pieces independently, because they are isolated in a separate process. You can run 5, 10 or 20 - as many as you need, product catalogs, user authorization blocks, or something else.
It turns out very small, in comparison with how it was previously designed, services.
This approach has several advantages , which we will talk about a little later, but in general this is not the topic of today's story.
In 2014-2015, I decided to look for whether there is a living experience in the Russian corporate sector using such architectures. Then there was nothing among my friends and colleagues - it was interesting to chat theoretically, but no one had practice.
In 2016, “something” began to be. In some banks, you could hear that "we are trying to use microservice architecture, we have Docker and more." It was interesting what came of it.
In 2017, we at CUSTIS decided to hold a meeting “Microservices for Enterprise” . It was a semi-closed event, mostly called familiar architects from Enterprise.
At this mitap, it was discovered that there is still a lot of misunderstanding, especially from IT managers. They do not understand why this is necessary at all. It seems to them that this is some kind of “another invention of programmers”, from which only unnecessary trouble. Therefore, I once again tried to look at this phenomenon from a point of view that, perhaps, would be more understandable not to techies, but to people in the position of IT director or development director. Well, if you are a technical specialist, perhaps this story will help you explain to the management the meaning and benefits of the microservice approach.
You can not stop for a long time at the fact that the interest in the network to the topic of microservices in recent years has grown tremendously.
Since 2014, several international conferences have been held specifically on microservices.
It turns out a huge amount of literature on this topic.
Information is just a storehouse, but, nevertheless, in the Russian corporate party, if you look at the total mass, they are just starting to talk about it. Of course, everything is heterogeneous - somewhere the practice of application is greater, somewhere less, somewhere quite neither sleep, nor spirit.
Therefore, I am trying to fill this gap.
My story will be built as three different development stories . Together we will try to look at this trend from three different angles. I think this is very useful for understanding nature.of what we are dealing with, and as a result, in order to draw the right conclusions and make the right decisions regarding the trend of microservices.
In the first story, we will go from the outside to the inside, as it should be in systems engineering, and we will talk about the market and about need - about Enterprise and the Web as two worlds .
In 2014, Gartner released an analytic report that said that by 2017, 50% of the main companies should definitely use the Web-scale architecture, but it was not at all clear what it was. When we decided to organize a Web-scale IT Conference, we tried to determine what Web-scale technologies are, what Enterprise is, and somehow see the difference.
Then for myself I drew about such a picture. The conditional “Web” and the conditional “Enterprise” are two large industries, or two large segments of the industry, from the point of view of IT today they look very different due to the fact that they had significantly different development paths:
Initially, these two worlds were almost completely divided. The Internet appeared as something isolated, and for 10-20 years experienced tremendous evolutionary pressure on this island of its own.
Clash of
What are the interests and fears of the classical business that seem to be justified? I like the metaphor of two colliding continents.
There was the mainland Enterprise - the market for classic services with automation, with its unhurried evolution. And suddenly it turns out that another continent is going into a collision - another world, in which the evolutionary process, too, is at its own pace. And it may turn out that for us, the inhabitants of the familiar Enterprise world, the results of this evolutionary process will not be very friendly - as in the picture.
We already see how Google deals with automobiles, Airbnb - hotels, Amazon - offline-shops, etc.
Thus, it turns out that if we, as the IT departments of corporations or as contractors of the Enterprise world, do not pay attention to these factors in any way, then we can regret it very much.
Analyzing the technical side of how the Internet giants work, they name a lot of different patterns and mechanisms that are part of the Web-scale architecture.
In my opinion, microservices are one of the most important, complex, and significant components that have the greatest consequences for redesigning the device of techniques and patterns in Enterprise.
Of course, like any innovation, the topic of microservices cannot escape its hype period. There are companies that will happily come running and introduce for money - so everything is twisted, and high expectations arise.
It is difficult now to say in which area the technology itself is now located - maybe at the highest peak of expectations, and we still have to give up on it ten times more. But, nevertheless, this is not a reason not to study it right now .
Summing up the first story about markets and needs:
These are the benefits that the microservice architecture can potentially bring to Enterprise. If in some area of your business you intend or think that you have to compete with Internet companies, it may very well be that you need to borrow their technology and try to apply it in these areas yourself, build competencies and set the same tasks - increase loads and variability. Otherwise, then just eat.
3. They are already here.
If you yourself are from the Enterprise world, then you see how vibrant, dynamic and unlike what we are dealing with in a classic enterprise.
We again go outside - inside: from the market, that is, from the environment of a separate enterprise, we move on to the installation of equipment in this separate enterprise. Let's talk about how the architectural styles of enterprise software have changed, how IT was generally built in the enterprise.
Like any systems and patterns, the development of IT in an enterprise goes from problem to problem. We solve one problem due to a certain pattern and often bring some other problems, this is an inevitable path. Moreover, we always move from simpler to more complex , these are general laws of development.
At one time, my colleagues and I were discussing - how is it that we always want to make it easier for people. Why, on the contrary, is everything getting complicated and complicated?
That is, something that had to be fought for with a large number of work hours, at some point we simply begin to receive, as an infrastructure , sometimes indifferent to our applied tasks. This “sediment” is increasing all the time, but the complexity of the systems only grows with time.
Once upon a time (probably you, like me, did not find these times), all IT in enterprises was done on one large computer. From the infrastructure there was only hardware, an operating system and files. Everything else that is needed for calculations (data storage, calculation logic, command line interface, etc.), our predecessors had to write with their hands.
Time passed, IT capabilities changed, and now on every table there is a personal computer, common data is somehow connected to the network and stored on a shared file server. More opportunities appear in the infrastructure: we don’t think of how to access the network ourselves, there are tools for working with files on a file server, the first databases and more.
But of course, we still write more complex tasks with our hands: we need to store something in the database; still need to calculate the business logic; make an already more complex graphical interface and more.
The next step is relational databases. There was a long period of time when IT in enterprises was built on a client-server architectureas an architectural style. This was supported by all vendors: here is the database, here is a quick development tool.
Here is part of the complexity of what we wrote with our hands at the previous step, “falls into some sediment”, into the infrastructure. We get it out of the box, but there is always enough work. We are writing data schemes of an increasingly complex structure, query languages are more abstract, SQL has appeared. We develop business logic, user interfaces, etc.
After some time, three-tier architecturecaptured the minds and interests of corporate software developers. For display, for example, there are slightly more blanks in the form of a browser and a library of components. Nevertheless, a lot of manual labor remains, a huge number of integration tasks arise. The application server is largely involved in integration, but it is easier to scale.
This process continues - service-oriented architecture . Corporate IT, at some point begins to go on the Web, even in B2B systems, specialized Web servers and corporate buses appear to integrate many solutions among themselves, systems for orchestrating complex processes of interaction between all these systems. And each time the story repeats itself - we get something out of the box, but we still add a lot of things with our hands.
After such a retrospective, one can notice that each time we have more and more ambitious tasks, their complexity and non-functional requirements are growing. As a result, IT systems begin to granulate:
It is easy to assume that this process does not stop suddenly on three-tier and service-oriented architectures. My hypothesis is that it continues exactly in microservice architecture.
What do we have at the moment?
The variety of services included even in one application is amazing and even scary.
The common services that different applications need are now provided from the cloud, for example, there you can find authorization forms. It blows up the brain a little, but it follows the same principles of development - it stands out, specializes and belongs to some infrastructure .
A huge number of opportunities that previously had to be done with your hands for each responsible system, now more and more often you want to do universally for any services. Again, I really want it to be provided “out of the box” - everything related to automatic scaling, system monitoring, high availability, logging, which must already be done centralized, because otherwise it’s impossible to climb tens and hundreds of machines to understand what happened.
That is, at the same time there is a huge demand for infrastructure , and slowly, with delay, the market of vendors providing it is being pulled up.
In fact, it is amazing how many infrastructure tasks arise. This is really a huge amount of work, for example, Avitosays that making a monitoring system for microservices is 1 person-year of work.
I think that in about five years we will still see good infrastructures for supporting microservices out of the box from different manufacturers. But for now, it's still raw enough. Although without it, nowhere else - the infrastructure is becoming thicker, services and functions are becoming more granular .
An interesting moment is taking place. When now users come to you and start interacting with a particular device, it’s even hard to say what an “application” is. Previously, there were different applications in the enterprise, and their boundaries were clearly understood.
Now the user logs in from one device, and then from another - is this the same application or not? We look inland - there are some App Gw, which in turn interact with different services. Where is the application here? There is a feeling that after some time the concept of the application can become very blurred .
Perhaps only the boundaries of business functions will somehow restrain this confusion. And then, in the presence of a large number of infrastructure services, it is foolish to produce them. Everything that in your systems will not be tied strictly to business functions, apparently, will be done in the general infrastructure of the enterprise. But now this is not so, and in the next 5-10 years it will still not be so, probably.
A very interesting point in this entire process - but do you, a specific medium or even large enterprise, need all of this — microservices and the accompanying complication? Maybe you can wait and not go up this mountain?
I think, fortunately or unfortunately, but jumping out of this process will not work . This is a trend that is stronger than us, and carries all. We have a choice - either try to wait aside, or still integrate.
The trouble is that emerging new infrastructure styles across the enterprise are beginning to push you towards new architectures, even if your enterprise specifically does not need it today. For example, you do not need super scalable systems and big data, you have a relatively small business that works great. With this, perhaps, in fact, a two-tier application on Delphi is calmly coping.
But the important thing is that the whole focus of attentionvendors and researchers who develop technology, is now directed here - at this point. It seems that this does not directly concern you, but it turns out that the whole industry is developing here.
The vector of aspiration of your employees is also directed in this direction, because it is fashionable, it is a hype, it is books, conferences and everything else. If you don’t try to somehow integrate into this movement, after a while you realize that you will have to work with very specific frames, which the new one does not care about at all. I am not saying that these are bad workers. But some very good people will be washed out of you if you do not at least to some extent follow the trend.
Therefore, I believe that sooner or later it will be necessary to infuse, even if you really don’t feel like it, so it’s better to do it in a managed way within your enterprise. That is, to lead, if it is already impossible to fight .
Summarizing the history of the evolution of architectural styles in the enterprise, we can record the following:
And we go even deeper. We talked about markets and why it is worth thinking about it, and how it fits into the general trend of Web-scale. Now I want to talk about how the roles of architects and their specialization will change in connection with the trend of microservice architecture.
Once upon a time, software development was handled by a development team and a maximum of one manager-manager who managed both work and resources. If the system was large, then it suddenly became clear that if you did not care about architecture (and there was no such word in relation to software before) and did not manage it, then it would grow “somehow by itself”, and then, stiffened, it would begin to manage your project in your place.
It became clear that the position of the architect is still needed , I repeat - on relatively large systems.
Within a large enterprise, it is still a little more complicated.
At first they managed with some separate independent self-written systems. Then it suddenly became clear that there are a lot of such systems, often they are delivered by the vendors already ready, and they need to be integrated with each other. Here, the profession of just a software architect is no longer very necessary, but such a specific position of a person is needed who would implement the following system, complete something small and tie it all together. The position of the integration architect (specialist in integration solutions) appears and the concept of solution arises .
This is an interesting specialization when a solution architect can be a person who has grown out of analysts and does not know how to program individual software, but performs a large, complex and useful work.
It is very interesting how during this period, before microservices, work is usually carried out with infrastructure at the enterprise. It is almost completely fenced off, individuals at their own pace, in their life cycle of changes, provide a uniform, standard and very stable infrastructure for all enterprise applications:
This is the philosophy of IT services in large enterprises - for many years everything has been aimed at increasing work efficiency, and therefore everything is standardized - ordering something unusual is simply impossible.
This is also some kind of management system - we separately manage applications and separately - infrastructure, and for each part there are separate bosses.
In this picture, which has been kept for a long time at enterprises, unpleasant moments arise:
First. It is very difficult to bring new technologies , because they work in the infrastructure unit, trying to reduce costs and make everything standard.
And the second one. It’s more profitable for a vendor to bring you as large a product as possible, which has “everything you need for your business”. And thenthe functional architecture of the applications in your enterprise will ultimately be driven by several large vendors , not you.
The functions of your application will be what the vendors came up with. Or you need to spend a large amount of effort, time and a precious managerial resource on building your own IT.
The parity is very unpleasant for some market players. If you are a second-tier business, this may suit you. If you are trying to work in leadership positions, you need to compete, which is very hindered by the fact that any template changes are a terrible pain.
If the trend of microservices really develops as one can assume, we will see that large products will be divided into a fairly large number of separate services managed quasi-independently, approximately, as in the following picture.
Of course, there will remain the concept of a solution that describes a holistic assembly of several services aimed at solving one problem.
The position of the integration architect will, of course, remain, but perhaps it will shift more towards technical architecture. If now often the integration architect also deals with information architecture at the enterprise, understands what data goes where, then in the future, most likely, these functions will be divided, because the amount of work will increase - the more microservices, the more integration work.
Of course, the position of the architect of a separate service will remain. She will not go anywhere - both from vendors and from the enterprise. Moreover, in my opinion, the demand for it will increase, on the one hand, because there are more and more “cubes”, and on the other hand, ensuring this position will become a little easier, as the “cubes” become smaller.
Previously, to make the architecture of a large monolithic ERP system, a person with simply colossal experience was needed. In a situation where services are more isolated from each other, and the microservice architecture outlines their boundaries and defines contracts, this position will become easier to provide. People with less qualifications will be able to replenish this need . In my opinion, this is great.
Strong changes will occur in the management of infrastructure, in what is now often heard the term DevOps . Many say that this is a connection between programmers and administrators. It seems to me that this does not capture the essence. In fact, the task itself has changed: it’s not just someone else who provides the entire infrastructure, and it works stably and equally. For each such decision, for each service, something different is now required - its own databases, different monitoring tools, and so on. There is a huge need for infrastructure that does not come out of the box , but it must be assembled by hand and often another, new, different one.
That is, in principle, the nature of this work and the nature of managing people are changing. You can no longer manage this over TSM. You often need to manage this in project form, because the project includes a huge piece of completely unique infrastructure for a specific solution.
Therefore, DevOps appears as an infrastructure developer , and an infrastructure architect may appear nearby .
Interestingly, recently I sometimes see that the architect is also called the person who, in this situation, is called a boiling variety of technologies and says:
That is, it is not even a software architect in the full sense. This is a specialist in the technological framework, one might say, a technologist. Its task is to understand and make a wide variety of raw open source technologies work. Apparently, such specialization also stands out now, maybe not in the form of a full-time employee, but the person who came, helped, got started and further advises.
I will briefly summarize the entire report. I tried to give an idea of how the MSA trend looks when viewed “from the Enterprise world” from three sides and in historical context from the past to the future, in terms of:
It seems to me that such a voluminous vision will help CIOs better position themselves in the MSA trend and understand what to do with a particular company - to shut itself off or, conversely, to integrate.
news
Together with Igor Bespalchuk we will try to look at this trend from three different angles, which is very useful for understanding the nature of what we are dealing with, and, as a result, in order to draw the right conclusions and make the right decision.
Microservices are one of the most important and significant components of the Web-scale architecture, which has the greatest consequences for redesigning the device of techniques and patterns in Enterprise. It is difficult now to say in which area the technology itself is now located - maybe at the highest peak, and we still have to give up ten more times. But, nevertheless, this is not a reason not to study it right now .
About the speaker: Igor Bespalchuk - application architect at the CUSTIS group of companies, which develops custom software for large corporate parties and government agencies. Prior to this, Igor also worked with corporate companies and retail, so he had the opportunity to observe how development processes are taking place in the corporate sector and how the management system works, as well as how certain changes are gradually leaking there.
Next is the transcript of Igor's report at the Web-Scale IT Conference 2017.
My introduction to the MSA theme
I first came across microservice architecture in November 2012 when I came across a wonderful presentation by James Lewis with QCon "Services: Java, the Unix Way" . He talked about how they worked on a large, complex, loaded project with a huge amount of data: they sawed it into separate functional elements, and decided to implement them as separate services. It was so amazing and unusual, including for our practice of architectural design, that directly excite the imagination. And at the same time, this intersected strongly with the concepts of Domain Driven Design (DDD), which we used quite deeply in our practice for many years, but in monolithic systems.
And in 2014, on the website of Martin Fowlera large article “Microservices” appeared , where he clarified what microservices are and why they are needed, the article is translated into Russian .
In a nutshell, we used to do heterogeneous functionality together, there was one system or application, and it was executed as a single process. If it was necessary to scale the system, this was done by horizontal scaling, but uniformly throughout the functionality.
In short, the central idea of microservices is that we begin to put a separate type of functionality in a separate process - Bounded Contextas it is called in DDD. If necessary, we can scale these pieces independently, because they are isolated in a separate process. You can run 5, 10 or 20 - as many as you need, product catalogs, user authorization blocks, or something else.
It turns out very small, in comparison with how it was previously designed, services.
This approach has several advantages , which we will talk about a little later, but in general this is not the topic of today's story.
In 2014-2015, I decided to look for whether there is a living experience in the Russian corporate sector using such architectures. Then there was nothing among my friends and colleagues - it was interesting to chat theoretically, but no one had practice.
In 2016, “something” began to be. In some banks, you could hear that "we are trying to use microservice architecture, we have Docker and more." It was interesting what came of it.
In 2017, we at CUSTIS decided to hold a meeting “Microservices for Enterprise” . It was a semi-closed event, mostly called familiar architects from Enterprise.
At this mitap, it was discovered that there is still a lot of misunderstanding, especially from IT managers. They do not understand why this is necessary at all. It seems to them that this is some kind of “another invention of programmers”, from which only unnecessary trouble. Therefore, I once again tried to look at this phenomenon from a point of view that, perhaps, would be more understandable not to techies, but to people in the position of IT director or development director. Well, if you are a technical specialist, perhaps this story will help you explain to the management the meaning and benefits of the microservice approach.
Online interest
You can not stop for a long time at the fact that the interest in the network to the topic of microservices in recent years has grown tremendously.
Since 2014, several international conferences have been held specifically on microservices.
It turns out a huge amount of literature on this topic.
Information is just a storehouse, but, nevertheless, in the Russian corporate party, if you look at the total mass, they are just starting to talk about it. Of course, everything is heterogeneous - somewhere the practice of application is greater, somewhere less, somewhere quite neither sleep, nor spirit.
Therefore, I am trying to fill this gap.
My story will be built as three different development stories . Together we will try to look at this trend from three different angles. I think this is very useful for understanding nature.of what we are dealing with, and as a result, in order to draw the right conclusions and make the right decisions regarding the trend of microservices.
The first story. Enterprise and Web as two worlds
In the first story, we will go from the outside to the inside, as it should be in systems engineering, and we will talk about the market and about need - about Enterprise and the Web as two worlds .
In 2014, Gartner released an analytic report that said that by 2017, 50% of the main companies should definitely use the Web-scale architecture, but it was not at all clear what it was. When we decided to organize a Web-scale IT Conference, we tried to determine what Web-scale technologies are, what Enterprise is, and somehow see the difference.
Ways of development
Then for myself I drew about such a picture. The conditional “Web” and the conditional “Enterprise” are two large industries, or two large segments of the industry, from the point of view of IT today they look very different due to the fact that they had significantly different development paths:
- Enterprise companies have evolved from a classic business, which mainly provides tangible goods and services, and IT exists as a means of automating this business.
- On the Web, from the very beginning, both the product and the service often had a digital form, or, as in the case of Amazon, a very significant share in the service itself is digital.
Initially, these two worlds were almost completely divided. The Internet appeared as something isolated, and for 10-20 years experienced tremendous evolutionary pressure on this island of its own.
Web evolutionary pressure
- Lack of physical growth restrictions
- The explosive growth of new types of services , including digital.
- Fierce competition for an unlimited volume of customers , which covers the whole world, regardless of territorial location.
- Requirements for UI / UX, load and scaling, development.
- A frequent change of technology, therefore, a stable homogeneous infrastructure and architectural style does not have time to form.
- The wave of development of Open Source, the cult of a heavy vendor is not formed.
- Result: some survived by spawning a series of technical and organizational patterns that meet these requirements.
Clash of mainland markets
What are the interests and fears of the classical business that seem to be justified? I like the metaphor of two colliding continents.
There was the mainland Enterprise - the market for classic services with automation, with its unhurried evolution. And suddenly it turns out that another continent is going into a collision - another world, in which the evolutionary process, too, is at its own pace. And it may turn out that for us, the inhabitants of the familiar Enterprise world, the results of this evolutionary process will not be very friendly - as in the picture.
We already see how Google deals with automobiles, Airbnb - hotels, Amazon - offline-shops, etc.
Thus, it turns out that if we, as the IT departments of corporations or as contractors of the Enterprise world, do not pay attention to these factors in any way, then we can regret it very much.
Analyzing the technical side of how the Internet giants work, they name a lot of different patterns and mechanisms that are part of the Web-scale architecture.
In my opinion, microservices are one of the most important, complex, and significant components that have the greatest consequences for redesigning the device of techniques and patterns in Enterprise.
Summary of the first story
Of course, like any innovation, the topic of microservices cannot escape its hype period. There are companies that will happily come running and introduce for money - so everything is twisted, and high expectations arise.
It is difficult now to say in which area the technology itself is now located - maybe at the highest peak of expectations, and we still have to give up on it ten times more. But, nevertheless, this is not a reason not to study it right now .
Summing up the first story about markets and needs:
- MSA is one of the technical patterns that emerged in the process of fiercely competitive development in the “parallel world” of Web technologies.
- In many ways, in this "parallel world" those who learned to provide:
- Retention of an online client with a high quality service
- high loads and data volumes that are completely not typical for Enterprise;
- rapid variability in technology , independence from vendors.
These are the benefits that the microservice architecture can potentially bring to Enterprise. If in some area of your business you intend or think that you have to compete with Internet companies, it may very well be that you need to borrow their technology and try to apply it in these areas yourself, build competencies and set the same tasks - increase loads and variability. Otherwise, then just eat.
3. They are already here.
If you yourself are from the Enterprise world, then you see how vibrant, dynamic and unlike what we are dealing with in a classic enterprise.
The second story. Enterprise software architectural styles
We again go outside - inside: from the market, that is, from the environment of a separate enterprise, we move on to the installation of equipment in this separate enterprise. Let's talk about how the architectural styles of enterprise software have changed, how IT was generally built in the enterprise.
The development of architectural styles
Like any systems and patterns, the development of IT in an enterprise goes from problem to problem. We solve one problem due to a certain pattern and often bring some other problems, this is an inevitable path. Moreover, we always move from simpler to more complex , these are general laws of development.
At one time, my colleagues and I were discussing - how is it that we always want to make it easier for people. Why, on the contrary, is everything getting complicated and complicated?
An understanding was born that the complexity of systems, including technical ones, never decreases , as it may sometimes seem - it “ precipitates ” in the form of infrastructure.
That is, something that had to be fought for with a large number of work hours, at some point we simply begin to receive, as an infrastructure , sometimes indifferent to our applied tasks. This “sediment” is increasing all the time, but the complexity of the systems only grows with time.
Once upon a time (probably you, like me, did not find these times), all IT in enterprises was done on one large computer. From the infrastructure there was only hardware, an operating system and files. Everything else that is needed for calculations (data storage, calculation logic, command line interface, etc.), our predecessors had to write with their hands.
Time passed, IT capabilities changed, and now on every table there is a personal computer, common data is somehow connected to the network and stored on a shared file server. More opportunities appear in the infrastructure: we don’t think of how to access the network ourselves, there are tools for working with files on a file server, the first databases and more.
But of course, we still write more complex tasks with our hands: we need to store something in the database; still need to calculate the business logic; make an already more complex graphical interface and more.
The next step is relational databases. There was a long period of time when IT in enterprises was built on a client-server architectureas an architectural style. This was supported by all vendors: here is the database, here is a quick development tool.
Here is part of the complexity of what we wrote with our hands at the previous step, “falls into some sediment”, into the infrastructure. We get it out of the box, but there is always enough work. We are writing data schemes of an increasingly complex structure, query languages are more abstract, SQL has appeared. We develop business logic, user interfaces, etc.
After some time, three-tier architecturecaptured the minds and interests of corporate software developers. For display, for example, there are slightly more blanks in the form of a browser and a library of components. Nevertheless, a lot of manual labor remains, a huge number of integration tasks arise. The application server is largely involved in integration, but it is easier to scale.
This process continues - service-oriented architecture . Corporate IT, at some point begins to go on the Web, even in B2B systems, specialized Web servers and corporate buses appear to integrate many solutions among themselves, systems for orchestrating complex processes of interaction between all these systems. And each time the story repeats itself - we get something out of the box, but we still add a lot of things with our hands.
Separation of functions
After such a retrospective, one can notice that each time we have more and more ambitious tasks, their complexity and non-functional requirements are growing. As a result, IT systems begin to granulate:
- Decentralization is becoming profitable.
- Improving the autonomy of individual functions. If one client computer crashes, the database remains in order.
- Performance scaling. Inelastic systems that cannot be easily scaled are no longer happy with anyone.
- Specialization of computing functions.
- Integration of the shared . A separate specialization arises in the integration of all the numerously cut pieces.
It is easy to assume that this process does not stop suddenly on three-tier and service-oriented architectures. My hypothesis is that it continues exactly in microservice architecture.
What do we have at the moment?
- Clients for whom we fight, and employees with increased usability requirements, come with different devices (browsers, mobile devices, etc.)
- The application services in which we implement business tasks still remain.
- Databases are specialized and shared. We are no longer satisfied with one universal database, which does everything badly. At a minimum, we want one to be very fast, and the other to be very large.
- Specialized nodes appear in order to adapt the application to different platforms ( Application-level gateway ).
- Developments related to orchestration and messaging are still needed.
The variety of services included even in one application is amazing and even scary.
The common services that different applications need are now provided from the cloud, for example, there you can find authorization forms. It blows up the brain a little, but it follows the same principles of development - it stands out, specializes and belongs to some infrastructure .
A huge number of opportunities that previously had to be done with your hands for each responsible system, now more and more often you want to do universally for any services. Again, I really want it to be provided “out of the box” - everything related to automatic scaling, system monitoring, high availability, logging, which must already be done centralized, because otherwise it’s impossible to climb tens and hundreds of machines to understand what happened.
That is, at the same time there is a huge demand for infrastructure , and slowly, with delay, the market of vendors providing it is being pulled up.
In fact, it is amazing how many infrastructure tasks arise. This is really a huge amount of work, for example, Avitosays that making a monitoring system for microservices is 1 person-year of work.
I think that in about five years we will still see good infrastructures for supporting microservices out of the box from different manufacturers. But for now, it's still raw enough. Although without it, nowhere else - the infrastructure is becoming thicker, services and functions are becoming more granular .
An interesting moment is taking place. When now users come to you and start interacting with a particular device, it’s even hard to say what an “application” is. Previously, there were different applications in the enterprise, and their boundaries were clearly understood.
Now the user logs in from one device, and then from another - is this the same application or not? We look inland - there are some App Gw, which in turn interact with different services. Where is the application here? There is a feeling that after some time the concept of the application can become very blurred .
Perhaps only the boundaries of business functions will somehow restrain this confusion. And then, in the presence of a large number of infrastructure services, it is foolish to produce them. Everything that in your systems will not be tied strictly to business functions, apparently, will be done in the general infrastructure of the enterprise. But now this is not so, and in the next 5-10 years it will still not be so, probably.
A very interesting point in this entire process - but do you, a specific medium or even large enterprise, need all of this — microservices and the accompanying complication? Maybe you can wait and not go up this mountain?
Common boat problem
I think, fortunately or unfortunately, but jumping out of this process will not work . This is a trend that is stronger than us, and carries all. We have a choice - either try to wait aside, or still integrate.
The trouble is that emerging new infrastructure styles across the enterprise are beginning to push you towards new architectures, even if your enterprise specifically does not need it today. For example, you do not need super scalable systems and big data, you have a relatively small business that works great. With this, perhaps, in fact, a two-tier application on Delphi is calmly coping.
But the important thing is that the whole focus of attentionvendors and researchers who develop technology, is now directed here - at this point. It seems that this does not directly concern you, but it turns out that the whole industry is developing here.
The vector of aspiration of your employees is also directed in this direction, because it is fashionable, it is a hype, it is books, conferences and everything else. If you don’t try to somehow integrate into this movement, after a while you realize that you will have to work with very specific frames, which the new one does not care about at all. I am not saying that these are bad workers. But some very good people will be washed out of you if you do not at least to some extent follow the trend.
Therefore, I believe that sooner or later it will be necessary to infuse, even if you really don’t feel like it, so it’s better to do it in a managed way within your enterprise. That is, to lead, if it is already impossible to fight .
Summary of the second story
Summarizing the history of the evolution of architectural styles in the enterprise, we can record the following:
- MSA is the next step in the development of architectural styles of complex enterprise software systems.
- MSA continues the logic that was before - the general movement towards specialization, granularization and the allocation of common infrastructures.
- Like all previous steps, MSA solves some of the problems that arise in previous styles, and gives rise to a number of new ones. Nothing is possible to do with this and you have to live with it - master and once again go through them.
- Free breakfast, of course, does not happen!
История третья. Роль и специализации архитектора
And we go even deeper. We talked about markets and why it is worth thinking about it, and how it fits into the general trend of Web-scale. Now I want to talk about how the roles of architects and their specialization will change in connection with the trend of microservice architecture.
Once upon a time, software development was handled by a development team and a maximum of one manager-manager who managed both work and resources. If the system was large, then it suddenly became clear that if you did not care about architecture (and there was no such word in relation to software before) and did not manage it, then it would grow “somehow by itself”, and then, stiffened, it would begin to manage your project in your place.
It became clear that the position of the architect is still needed , I repeat - on relatively large systems.
Within a large enterprise, it is still a little more complicated.
At first they managed with some separate independent self-written systems. Then it suddenly became clear that there are a lot of such systems, often they are delivered by the vendors already ready, and they need to be integrated with each other. Here, the profession of just a software architect is no longer very necessary, but such a specific position of a person is needed who would implement the following system, complete something small and tie it all together. The position of the integration architect (specialist in integration solutions) appears and the concept of solution arises .
This is an interesting specialization when a solution architect can be a person who has grown out of analysts and does not know how to program individual software, but performs a large, complex and useful work.
It is very interesting how during this period, before microservices, work is usually carried out with infrastructure at the enterprise. It is almost completely fenced off, individuals at their own pace, in their life cycle of changes, provide a uniform, standard and very stable infrastructure for all enterprise applications:
- Here is the Oracle database for you, here are the virtual machines - work, and we all support this in order, in strict standards.
This is the philosophy of IT services in large enterprises - for many years everything has been aimed at increasing work efficiency, and therefore everything is standardized - ordering something unusual is simply impossible.
This is also some kind of management system - we separately manage applications and separately - infrastructure, and for each part there are separate bosses.
In this picture, which has been kept for a long time at enterprises, unpleasant moments arise:
First. It is very difficult to bring new technologies , because they work in the infrastructure unit, trying to reduce costs and make everything standard.
And the second one. It’s more profitable for a vendor to bring you as large a product as possible, which has “everything you need for your business”. And thenthe functional architecture of the applications in your enterprise will ultimately be driven by several large vendors , not you.
The functions of your application will be what the vendors came up with. Or you need to spend a large amount of effort, time and a precious managerial resource on building your own IT.
The parity is very unpleasant for some market players. If you are a second-tier business, this may suit you. If you are trying to work in leadership positions, you need to compete, which is very hindered by the fact that any template changes are a terrible pain.
If the trend of microservices really develops as one can assume, we will see that large products will be divided into a fairly large number of separate services managed quasi-independently, approximately, as in the following picture.
Of course, there will remain the concept of a solution that describes a holistic assembly of several services aimed at solving one problem.
The position of the integration architect will, of course, remain, but perhaps it will shift more towards technical architecture. If now often the integration architect also deals with information architecture at the enterprise, understands what data goes where, then in the future, most likely, these functions will be divided, because the amount of work will increase - the more microservices, the more integration work.
Of course, the position of the architect of a separate service will remain. She will not go anywhere - both from vendors and from the enterprise. Moreover, in my opinion, the demand for it will increase, on the one hand, because there are more and more “cubes”, and on the other hand, ensuring this position will become a little easier, as the “cubes” become smaller.
Previously, to make the architecture of a large monolithic ERP system, a person with simply colossal experience was needed. In a situation where services are more isolated from each other, and the microservice architecture outlines their boundaries and defines contracts, this position will become easier to provide. People with less qualifications will be able to replenish this need . In my opinion, this is great.
Strong changes will occur in the management of infrastructure, in what is now often heard the term DevOps . Many say that this is a connection between programmers and administrators. It seems to me that this does not capture the essence. In fact, the task itself has changed: it’s not just someone else who provides the entire infrastructure, and it works stably and equally. For each such decision, for each service, something different is now required - its own databases, different monitoring tools, and so on. There is a huge need for infrastructure that does not come out of the box , but it must be assembled by hand and often another, new, different one.
That is, in principle, the nature of this work and the nature of managing people are changing. You can no longer manage this over TSM. You often need to manage this in project form, because the project includes a huge piece of completely unique infrastructure for a specific solution.
Therefore, DevOps appears as an infrastructure developer , and an infrastructure architect may appear nearby .
Interestingly, recently I sometimes see that the architect is also called the person who, in this situation, is called a boiling variety of technologies and says:
- Tie us, please, all this together so that it “just starts working.” And then our applicants will understand.
That is, it is not even a software architect in the full sense. This is a specialist in the technological framework, one might say, a technologist. Its task is to understand and make a wide variety of raw open source technologies work. Apparently, such specialization also stands out now, maybe not in the form of a full-time employee, but the person who came, helped, got started and further advises.
conclusions
I will briefly summarize the entire report. I tried to give an idea of how the MSA trend looks when viewed “from the Enterprise world” from three sides and in historical context from the past to the future, in terms of:
- market needs in the worlds of Web and Enterprise, where does all this come from and what threatens us if we do not master it;
- architectural styles of enterprise software systems - and why you most likely will not be able to jump out of this boat;
- specializations of the role of architect , which are also intensively changing and sharing.
It seems to me that such a voluminous vision will help CIOs better position themselves in the MSA trend and understand what to do with a particular company - to shut itself off or, conversely, to integrate.
news
Фестиваль конференций Российские-интернет технологии уже не за горами, в этом году мы решили распределить доклады конференции на стыке enterprise и web-технологий Web-Scale IT по другим конференциям — технические доклады пойдут в Backend Conf РИТ++, управленческие в Whale Rider. На настоящий момент есть несколько интересных заявок, например:
- Готовность организации к изменениям или куда приводят мечты / Денис Кочергин (Ярмарка Мастеров)
- Как большая IT команда может меняться сразу по всем фронтам и при этом не умереть? / Александр Андронов (Додо Пицца)
- Кто живет на облаках или что такое cloud-native infrastructure / Антон Вайс (Отомато)
- Own cloud accessible to everyone: the life cycle of a typical microservice and its dependencies in kubernetes / Maxim Vikharev (Alytics)
The statuses of reports will be updated during April, but we promise to select the coolest ones, so it is not necessary to wait at all - you can already book tickets .