March 23, 2018 at 10:40How and why to move from a service-oriented architecture to microservices
Hello, my name is Alexey, I am the chief IT architect of Renaissance Credit Bank. About ten years ago, we, like many companies, accelerated their development thanks to service-oriented architecture (SOA). But over time, the requirements for architecture changed, and serious questions began to arise for this paradigm. In the end, we decided to move from an ESB integration bus to microservices. In our example, I will explain why it is worth thinking about the effectiveness of SOA and what can be done if this model also does not suit you.
It all started with the fact that we appreciated the structure of our integration bus. The diagram shows a typical situation with integration flows for ESB.
There can be several dozen such blocks (up to hundreds!), And they are closely intertwined. If you need to make a new product and make any changes, then no resources will be enough to process this hefty monolith. And this is just a general view. We studied SOA from all sides and identified seven main problems:
A traditional waterfall-oriented service tire with its complex and slow processes does not fit into the market. Today, customers are aware of agile development methodologies. They may not know the differences between agile and waterfall, but they want to get the first results - from the idea to the prototype in operation - not in 6-8, but in 2-3 months, but better in general in one. Let it be MVP, but it is important for the customer to check the business idea as soon as possible, to understand that the solution will take off.
In such conditions, traditional, horizontally oriented teams are inferior to vertically oriented, which work with all components of the product - from the interaction channel to the back end. The edge raises the question: what to do with the integration bus, which has turned into a monolith due to the reuse of services? We need a new technological approach.
We began to work out an alternative to SOA with the wording of the basic requirements:
In accordance with the requirements, we have formed a pyramid of technologies:
Now about the components of each level separately:
1. The level of methodology
2. Infrastructure level
3. Applied architecture
Please note: there is no mandatory reuse requirement for MSA !
4. The level of the data model
5. The level of integration.
Based on a new combination of technologies, we presented what we want to come to. The figure below is a diagram of our current infrastructure with an integration bus. Nearby is the scheme that we are striving for.
What is changing? Initially, at the back office level, any ABS has a business logic and data source. We strive to ensure that back-office systems are reduced exclusively to storage and “nuclear” functionality, where changes would be rare. And the product logic has gone to the level of microservices, which allow you to flexibly change and create products that are divided by domain.
On the scale of the channel, we plan on the basis of microservices to form and manage the logic “spread out” along the front-end layer. As a result, front-end applications themselves will contain only channel logic, that is, native applications for automating the necessary channels for servicing requests.
All access control will be carried out through the API Management portal, where all APIs will be described and published. Here, any developer can get all the information about them and join the development factory. With the help of GitLab technologies, a continuous cycle of work is organized in it - planning, development, testing, release and operation.
API Management and Open API Schema
Changes of this magnitude do not go without difficulties. They are mainly associated with breaking up monolithic box systems into microservices, as well as with restoring the connections in the black boxes of our ESB and Data Silo. With the model of microservices, the use of a BPM engine for building business processes is noticeably losing its relevance. And now the issue of its alternative - event-driven architecture and choreography is becoming very important for us. We also plan to move from pure DevOps to DevSecOps, which will include ITSec requirements, and split our Data Silo into domains. Performing these tasks will require actively collecting experience to test and maximize the use of the described concepts.
New ESB Issues
It all started with the fact that we appreciated the structure of our integration bus. The diagram shows a typical situation with integration flows for ESB.
There can be several dozen such blocks (up to hundreds!), And they are closely intertwined. If you need to make a new product and make any changes, then no resources will be enough to process this hefty monolith. And this is just a general view. We studied SOA from all sides and identified seven main problems:
- Emphasis on reuse of services. This rule was the cornerstone of SOA, but in the end it led to the fact that all teams (front, back and integration) were so tightly connected that any integration update required regression check of almost the entire bus!
- Heavy technology stack. At first, a single ESB container was an advantage, and this allowed us to quickly deploy services. But today, a limited stack of technologies, libraries and programming languages is already noticeably hindering development.
- Turn ESB into a bottleneck for all processes. Over time, the bus acquires functions and turns into a full-fledged information system of the bank, with a waterfall approach and the “cutting” of integration flows. The loading of the ESB team is increasing significantly, which slows down the work of the rest of the development teams.
- DataSilo. In SOA, service interfaces are separate from implementation. But how separated are the data used by the services? One service accesses the data of another, the dblink mechanism is used, there is confusion and the “data silo”.
- Mixing integration patterns. Strictly speaking, there are two for SOA: the classic MessageBroker, where messages are sent between systems for exchanging portions of data, and the hub of services, where we place some services that “have nowhere else to stick to”. When these templates are mixed, the ESB turns into a completely black box.
- The formation of "shadow IT". Business customers often form their own IT teams to develop critical systems for themselves, but cannot use the bus for integration. So the number of “left” connections to information systems is growing.
- Lack of support for ITSec (information security) at the concept level. Here you need to make a little clarification: firstly, the completely “western” concept of ESB does not take into account the peculiarities of Russian legislation (for example, the protection of personal data when transferring from source to receiver), and secondly, the ESB container is a perimeter that works effectively, when inside it is no longer required to comply with the special requirements of ITSec.
New customer needs
A traditional waterfall-oriented service tire with its complex and slow processes does not fit into the market. Today, customers are aware of agile development methodologies. They may not know the differences between agile and waterfall, but they want to get the first results - from the idea to the prototype in operation - not in 6-8, but in 2-3 months, but better in general in one. Let it be MVP, but it is important for the customer to check the business idea as soon as possible, to understand that the solution will take off.
In such conditions, traditional, horizontally oriented teams are inferior to vertically oriented, which work with all components of the product - from the interaction channel to the back end. The edge raises the question: what to do with the integration bus, which has turned into a monolith due to the reuse of services? We need a new technological approach.
The foundation of a new approach
We began to work out an alternative to SOA with the wording of the basic requirements:
- The ability to quickly pilot ideas. This will require a simple (regular, repeating) structure of connections between systems, rapid deployment and versioning.
- Support for agile development. Easy connection of new, vertical teams, access to the level of “development factory”, with automation of routine tasks.
- The presence of an ecosystem with development teams of different types : internal, external, partnership. Providing interfaces for external access, controlling this access and billing.
Technology pyramid
In accordance with the requirements, we have formed a pyramid of technologies:
Now about the components of each level separately:
1. The level of methodology
- The Agile . A flexible approach now causes a lot of emotions and opposing opinions (on the topic of applicability in organizations of various sizes). For ourselves, we have formulated the main thing: this approach is the basis for structuring requirements, rapid prototyping and creating MVP - testing product ideas.
- DevOps. This paradigm requires maximum automation and “blurring the boundaries” between the development and maintenance of systems. It avoids the loss of time in routine deployment and maintenance operations.
- Factory development . The movement of artifacts from the analytic stage to the deployment and operation of the created product should be continuous, without re-creating artifacts at each subsequent stage. For example, there should not be a situation where the service interface is first described in some format by the analyst, and then the developer manually creates the same interface again, but in a different tool.
2. Infrastructure level
- Containerization (Docker) . The only option to provide microservices at this level is to avoid a single bus container and use separate containers for each instance of the service. This means that the use of "heavy" application servers running a wide range of libraries at startup is also not suitable. The container should be as lightweight and configurable as possible to ensure that only the set of functionalities necessary for the service is launched. And from this point of view, Docker is great.
- Container Orchestra . Containers should be managed by a tool that solves the typical tasks of fault tolerance, balancing and unifying deployment / stop / start. Moreover, this tool cannot be an analogue of a tire with its monolithic container.
3. Applied architecture
- Microservice Architecture (MSA) . The question of the exact definition of microservice is still open. But for the full development of integration architectures and systems, we have identified the following key properties of microservices:
| Property | Explanation | |
| 1 | Separation of interface and service implementation | This property is inherited from SOA and requires that a change in implementation does not require a change in interface. A call to the service required only knowledge of the interface, without understanding the intricacies of implementing the service. |
| 2 | Good granularity | Microservices should be relatively small (the “two pizza rule”) and separated from each other so that changes in functionality in any subject area are concentrated in a maximum of one microservice. |
| 3 | Separation of services at all levels | Microservices should be completely separated from each other. At the interface level and at the execution level - each has its own execution container. At the data level, the microservice has access only to “its” data and does not know anything about the database features of the neighboring microservice. |
| 4 | Unified interaction | If the microservice needs data that other microservice provides access to, the interface call should work. In no case do not access through the database to the adjacent circuit. |
Please note: there is no mandatory reuse requirement for MSA !
4. The level of the data model
- DDD (Domain Driven Design) . One of the most difficult issues is the set of rules by which microservices are created, and ensuring their connection with the earlier stages of analytical development of software. We tried to build on the DDD concept with two main goals. First, it allows you to create domains in the subject area and successfully associate them with product teams (agile!). Secondly, it helps to form a microservice as an API for working with a specific business object - corresponding to a resource for a RESTful service.
- Domains correspond to bank products. This allows you to move away from the classic separation of the front, back and integration, to come to unity with product teams.
5. The level of integration.
- API Management and the First API Approach . Integration using a bus means “cutting” the integration flows from front to back with reuse of services. In the new version, we focus on the “API First” approach. Back-end systems prepare the most common APIs. Integration is based on the principle of the base crystal: front-end system developers select the API calls they need published on the API Management portal.
- The API the Open . The Open API implies the use of the API Management system for organizing access and operation of fundamentally different categories of developers - external, internal, partner ecosystem participants. In fact, we get a public organization API.
Architecture changes
Based on a new combination of technologies, we presented what we want to come to. The figure below is a diagram of our current infrastructure with an integration bus. Nearby is the scheme that we are striving for.
What is changing? Initially, at the back office level, any ABS has a business logic and data source. We strive to ensure that back-office systems are reduced exclusively to storage and “nuclear” functionality, where changes would be rare. And the product logic has gone to the level of microservices, which allow you to flexibly change and create products that are divided by domain.
On the scale of the channel, we plan on the basis of microservices to form and manage the logic “spread out” along the front-end layer. As a result, front-end applications themselves will contain only channel logic, that is, native applications for automating the necessary channels for servicing requests.
All access control will be carried out through the API Management portal, where all APIs will be described and published. Here, any developer can get all the information about them and join the development factory. With the help of GitLab technologies, a continuous cycle of work is organized in it - planning, development, testing, release and operation.
API Management and Open API Schema
What's next?
Changes of this magnitude do not go without difficulties. They are mainly associated with breaking up monolithic box systems into microservices, as well as with restoring the connections in the black boxes of our ESB and Data Silo. With the model of microservices, the use of a BPM engine for building business processes is noticeably losing its relevance. And now the issue of its alternative - event-driven architecture and choreography is becoming very important for us. We also plan to move from pure DevOps to DevSecOps, which will include ITSec requirements, and split our Data Silo into domains. Performing these tasks will require actively collecting experience to test and maximize the use of the described concepts.