July 2, 2018 at 18:49How bloody enterprise wins open-source: the battle for BPMS
The gears of a modern bank spin in accordance with financial business processes. They are more complicated than usual - this rule works for everything, to which you add the definition of “financial”. On the one hand, everything is complicated by regulators, countless approvals and parties involved. On the other - clumsy monolithic BPMS (Business Process Management System). In this post, we will tell you how you successfully abandoned one such system and went into a flexible and functional open source.
Programmers display business processes using different notations. Now the standard is BPMN (Business Process Model and Notation) - these are XML files with images attached to them. To work with this notation, BPMS products are created - monolithic proprietary systems that try to accommodate the maximum tools for developing BPM: from the user interface editor to the version control system.
Only developers who have been working with these systems for a long time can expand them. In enterprise BPMS, a REST API is provided, that is, systems can be accessed and received in response. But modifying and deep tuning BPMS itself is almost impossible. It is possible to work with such BPMS only through a limited set of tools by the manufacturer - a proprietary version control system, compiler, deployer - usually for each large BPMS a whole set is developed. These tools are developing slowly, the same problems may persist from release to release, because not so many people are involved in the work, as is the case with open source. In general, the capabilities of enterprise BMPS and the needs of their users coincide very rarely.
In the advertising of such systems, they talk about end-to-end workflow, they say that the business itself can change BPM processes on the fly. But in fact, even analysts cannot do this - only developed ones can handle it.
A business process consists of user and service tasks whose sequences lead to the final stop. In BPMS, through such schemes, you can track the execution time of business processes, as well as various business indicators - for example, KPI.
We often have to make changes of various sizes to business processes. We used to have a BPM process for client managers who sit in additional offices. In 2015, we closed some of the offices, and transferred managers to field work. This required big changes in BPM processes, it was necessary to introduce other roles, actions. Or, for example, the regulation of the economic security service has changed, and instead of two coordinators, now there are three.
First, we made changes through the IBM Lombardi BPMS. Having collected the typical flaws of the systems of its class, it was also distinguished by the lack of ordered documentation. It seemed that after the purchase of Lombardi Software, the software giant looked at the amorphous cloud of accompanying articles and decided not to touch anything. And even after reading all the documentation, it was impossible to do many things. For example, invoke a REST request with HTTPS authentication using a user certificate. Fortunately, the search for an alternative was a success.
After working with IBM BPM, we came to the conclusion that different user groups should be able to change business processes. Something insignificant in the online mode can be contributed by ordinary employees of business units. System analysts are changing the sequence of tasks in business processes. New integrations, changes in the UI and program code remain on the side of developers. And in order to maintain this flexibility, all BPMS must be deployed on microservices.
We can provide this approach with open-source BPMS Camunda. It is a fork of the Activity project, which the development team decided to make more selling. They put in order the documentation and began to develop Camunda separately, they earn on the sale of support.
BPMS Camunda allows you to edit business processes using standard Java and supports microservice sharing. Switching BPMS to microservices provides several advantages at once:
In the future, we plan to transfer one of our largest business processes to Camunda - corporate lending. Everything here is much more complicated than that of individuals and even than that of small / medium-sized businesses. It is not a product but a limited credit scheme that is applied, limits are limited in time, borrowers are usually part of larger organizations such as holdings, and holdings have something else to do. Limits are determined separately for each of these structures, and each structure has its own matching ones, the composition of which is constantly changing. We get a business process from hundreds of stages. We were able to change it with Camunda and decided to stay on it. Even if the developers decide now to close the project, the current capabilities of the system will last another three years.
Our first version of Camunda was based on Websphere and, in fact, was not much different from IBM Lombardi. We decided to write the applications that the service accessed on Spring Boot. They deployed on Tomcat and did not work on their own. Then it turned out that Camunda could work on Tomcat, a version for Spring Boot was developed. A full microservice architecture has already become available there. All applications with which the business process worked were implemented on a microservice architecture based on Spring Cloud.
Then it turned out that you can quickly change functionality not only in services that serve the business process. The BPM engine itself can be connected to any springboot application as a library.
Camunda as a microservice The
question arose: how many microservices should I raise? It was possible to make one server for each process, and there to place all the microservices, or for each task make your own microservice and in it a separate business process. The second would be more convenient, but it would be necessary to organize the interaction of processes scattered across individual microservices. We tried several options and settled on a solution when several microservices are responsible for a certain topic and several processes are grouped there. Communication takes place either through REST or through Rabbit MQ. Now they have launched a pilot on Kafka.
Business processes not only display business workflow, but also respond to events that occur in other systems. We have these events accumulated by a separate division of Big Data. Based on the analysis of these events, new business processes are created or existing ones are changed - this happens after the fact, with a frequency of, for example, once a day.
Ideally, business processes should move to changing online — for example, when demand for certain services increases, they will automatically prioritize their implementation and reallocate resources. This can be achieved through automation, interaction, for example, Kafka and Camunda, but this is a matter of at least several years. Perhaps now in the direction of changes in the online mode, the most fully advanced is the fully online English Monzo Bank. And we are also working on it.
Programmers display business processes using different notations. Now the standard is BPMN (Business Process Model and Notation) - these are XML files with images attached to them. To work with this notation, BPMS products are created - monolithic proprietary systems that try to accommodate the maximum tools for developing BPM: from the user interface editor to the version control system.
Only developers who have been working with these systems for a long time can expand them. In enterprise BPMS, a REST API is provided, that is, systems can be accessed and received in response. But modifying and deep tuning BPMS itself is almost impossible. It is possible to work with such BPMS only through a limited set of tools by the manufacturer - a proprietary version control system, compiler, deployer - usually for each large BPMS a whole set is developed. These tools are developing slowly, the same problems may persist from release to release, because not so many people are involved in the work, as is the case with open source. In general, the capabilities of enterprise BMPS and the needs of their users coincide very rarely.
In the advertising of such systems, they talk about end-to-end workflow, they say that the business itself can change BPM processes on the fly. But in fact, even analysts cannot do this - only developed ones can handle it.
A business process consists of user and service tasks whose sequences lead to the final stop. In BPMS, through such schemes, you can track the execution time of business processes, as well as various business indicators - for example, KPI.
We often have to make changes of various sizes to business processes. We used to have a BPM process for client managers who sit in additional offices. In 2015, we closed some of the offices, and transferred managers to field work. This required big changes in BPM processes, it was necessary to introduce other roles, actions. Or, for example, the regulation of the economic security service has changed, and instead of two coordinators, now there are three.
First, we made changes through the IBM Lombardi BPMS. Having collected the typical flaws of the systems of its class, it was also distinguished by the lack of ordered documentation. It seemed that after the purchase of Lombardi Software, the software giant looked at the amorphous cloud of accompanying articles and decided not to touch anything. And even after reading all the documentation, it was impossible to do many things. For example, invoke a REST request with HTTPS authentication using a user certificate. Fortunately, the search for an alternative was a success.
Camunda solves problems
After working with IBM BPM, we came to the conclusion that different user groups should be able to change business processes. Something insignificant in the online mode can be contributed by ordinary employees of business units. System analysts are changing the sequence of tasks in business processes. New integrations, changes in the UI and program code remain on the side of developers. And in order to maintain this flexibility, all BPMS must be deployed on microservices.
We can provide this approach with open-source BPMS Camunda. It is a fork of the Activity project, which the development team decided to make more selling. They put in order the documentation and began to develop Camunda separately, they earn on the sale of support.
BPMS Camunda allows you to edit business processes using standard Java and supports microservice sharing. Switching BPMS to microservices provides several advantages at once:
- Getting rid of the monolith. We can divide business processes by segments, linking them together using Rabbit MQ or Kafka through the queue. Business processes can be changed in isolation from each other, roll changes without waiting for a full release.
- Getting rid of a business process server.
- Scaling. If under load any server starts to fall, then it can be cloned. At peak loads, you can easily increase productivity by running multiple instances of business processes in different services.
- Versioning If, for example, you need to update the Java version, you can pick up several microservices with different versions of Java and run the new version in parallel. Each microservice can have not only different versions of software, but also different languages.
In the future, we plan to transfer one of our largest business processes to Camunda - corporate lending. Everything here is much more complicated than that of individuals and even than that of small / medium-sized businesses. It is not a product but a limited credit scheme that is applied, limits are limited in time, borrowers are usually part of larger organizations such as holdings, and holdings have something else to do. Limits are determined separately for each of these structures, and each structure has its own matching ones, the composition of which is constantly changing. We get a business process from hundreds of stages. We were able to change it with Camunda and decided to stay on it. Even if the developers decide now to close the project, the current capabilities of the system will last another three years.
Our first version of Camunda was based on Websphere and, in fact, was not much different from IBM Lombardi. We decided to write the applications that the service accessed on Spring Boot. They deployed on Tomcat and did not work on their own. Then it turned out that Camunda could work on Tomcat, a version for Spring Boot was developed. A full microservice architecture has already become available there. All applications with which the business process worked were implemented on a microservice architecture based on Spring Cloud.
Then it turned out that you can quickly change functionality not only in services that serve the business process. The BPM engine itself can be connected to any springboot application as a library.
question arose: how many microservices should I raise? It was possible to make one server for each process, and there to place all the microservices, or for each task make your own microservice and in it a separate business process. The second would be more convenient, but it would be necessary to organize the interaction of processes scattered across individual microservices. We tried several options and settled on a solution when several microservices are responsible for a certain topic and several processes are grouped there. Communication takes place either through REST or through Rabbit MQ. Now they have launched a pilot on Kafka.
Prospects for BPMS
Business processes not only display business workflow, but also respond to events that occur in other systems. We have these events accumulated by a separate division of Big Data. Based on the analysis of these events, new business processes are created or existing ones are changed - this happens after the fact, with a frequency of, for example, once a day.
Ideally, business processes should move to changing online — for example, when demand for certain services increases, they will automatically prioritize their implementation and reallocate resources. This can be achieved through automation, interaction, for example, Kafka and Camunda, but this is a matter of at least several years. Perhaps now in the direction of changes in the online mode, the most fully advanced is the fully online English Monzo Bank. And we are also working on it.