Experience in building an integration platform based on ServiceMix (Camel) and RabbitMQ

As soon as at least two information systems appear in the company that need to exchange data, the question arises of how to organize their interaction. There are many options: file exchange, links between databases, web or rest services, various messaging systems, outdated RPC and CORBA, new-fashioned gRPC, etc. The choice depends on the preferences of the project participants and on the capabilities of the systems (system architecture, platform used, availability of a ready-made API, etc.). Suppose you choose some kind of exchange, the systems began to interact, everything is fine. But then a third system arises, with which it is also necessary to integrate, then a fourth, etc. It’s necessary to sit down again and choose a method of exchange, and not the fact that it will be possible to limit oneself to the technologies already used (somewhere this is dictated by the limitations of new systems, somewhere, the developer insisted on another technology or wanted to try something new). As the number of systems grows, the number and complexity of interactions between them grows, and the number of technologies used increases. As a result, the entire integration architecture of the company begins to resemble a tangled ball of multi-colored threads, somehow linking the company’s systems, which is becoming increasingly difficult to unravel when parsing errors and improvements. Sooner or later, thoughts begin to come about creating a single integration environment that will transparently and expandably link all systems together. somehow linking the company’s systems, which is becoming increasingly difficult to unravel when parsing errors and improvements. Sooner or later, thoughts begin to come about creating a single integration environment that will transparently and expandably link all systems together. somehow linking the company’s systems, which is becoming increasingly difficult to unravel when parsing errors and improvements. Sooner or later, thoughts begin to come about creating a single integration environment that will transparently and expandably link all systems together.

In this article, I will describe the experience of using Apache ServiceMix (Camel) and RabbitMQ to build such an integration environment.

Technology stack

Rabbitmq

RabbitMQ is a messaging system based on the AMQP protocol. I will not describe the system in detail, on Habré there are already several articles that describe the system’s functionality in detail, I’ll talk about where and how we use RabbitMQ.



Through RabbitMQ we exchange data with the company's information systems. A set of queues is created for each system. For example, you need to organize a search for customers by name and date of birth in the accounting system. We create a couple of queues. One incoming in relation to the accounting system, we will send requests to it indicating the name and date of birth of the client. The accounting system listens for the incoming queue and processes incoming requests. The second is outgoing, to which the accounting system will send responses with a list of clients found that match the conditions of the request.

Why it is convenient:

• Messages that are stored in the queue are persistent. Those. if the system is unavailable for some time, then the messages will not disappear anywhere and will be processed after the system rises. Restarting RabbitMQ itself also does not result in message loss.
• The queue serves as a buffer that the system can process in a comfortable mode, avoiding peak loads. It is clear that for requests that the user is waiting on the interface, this will not work. In this case, an immediate response is needed. But for all kinds of asynchronous interactions it fits very well.
• Asynchronous interaction mode. At the same time, synchronous calls are also supported.

To teach the system to interact with RabbitMQ is not difficult. RabbitMQ has ready-made clients for various platforms (Java, .Net, Python, etc.). The client is simple and straightforward. Code that reads messages from the queue and / or sends messages to the queue takes several lines. It is clear that not every system can be made friends with RabbitMQ, for example, in the case of Legacy and boxed systems, this will be difficult. In this case, the exchange is built using technologies that support these systems, somewhere we call stored procedures in the database, somewhere we use web and rest services, somewhere else. In order to organize such interaction for many other integration tasks, the Apache ServiceMix product is used.

Apache ServiceMix

ServiceMix - Karaf container with a predefined set of bundles that are useful for solving various integration problems.



A little more detail about what is included in the product:

1. Actually, the Karaf container itself, in which bundles work and which allows you to manage them: install / uninstall / stop / start, view logs, see component dependencies, etc.
2. A wide range of bundles that perform classic integration functions: validation, various transformations (for example, from JSON to XML, transformations using XSLT, etc.), enrichment, routing, split and join, monitoring, execution of integration processes, etc. .
3. A wide range of different adapters: file adapters, adapters for web and rest services, JMS, RabbitMQ, Kafka, etc. A complete list of adapters can be found on the Camel website.

It is unlikely to be limited to using only pre-installed bundles, but it should be enough to start the installed set. Because Since we work with a Karaf container, you can install any necessary bundles, you can do this either by installing features (bundle sets), or simply by installing individual bundles. Of course, you can write your own bundles, or wrap third-party java libraries into bundles.

Apache camel

A key component of ServiceMix is ​​the Apache Camel framework, which allows you to build integration processes that are called routes in Camel terminology.
Let me show you an example - what is a route:



This is an example of a simple route that converts incoming multi-format messages to a common output format. Depending on the format of the message, the route routes it to the appropriate transformation, which converts the message to a common format, and then sends the result to the output.

Camel supports various notations for the description of routes, the most common Java DSL and Spring XML. We use Spring XML. Here's what the route in the image would look like in Spring XML notation:

/*[local-name()='Format_1']/*[local-name()='Format_2']/*[local-name()='Format_3']

A very nice addition is that Camel is perfectly integrated with Spring. You can use the familiar Spring XML, which defines beans, and in the same Spring XML define Camel routes. At the same time, bins can be called from routes, and routes can be called from bins. Calling bin from routes is implemented with Camel-specific flexibility, you can pass the message body to the bean method, you can pass headers + body, or you can only pass the value of a certain XML message tag specified via XPATH expression, and use the result of the method in the choice construct for further determination route. And all this is practically on one line.

Here is an example of Camel-style elegance:

${bean:authManager?method=checkToken(${body})}

public class AuthManager {
	public boolean checkToken(@Body Document xml, @XPath("/Root/Token/@Value") String token) {
		return checkSessionToken(token);
	}
}

Еще одно важное понятие в Camel – это endpoint (далее эндпоинт). Роуты могут читать сообщения из эндпоинтов, могут посылать сообщения в эндпоинты. Эндпоинтом может выступать, например, RabbitMQ очередь, файл в директории, опубликованный rest сервис и т.д. Если роут читает эндпоинт, то при поступлении сообщения в этот эндпоинт роут начинает его обработку. Это позволяет опубликовать роут, т.е. дать возможность обращаться к нему внешним системам. Если у вас есть роут, который выполняет какую-то задачу, например, проверяет корректность заполненных в анкете данных, то вы его можете опубликовать как web сервис, или дать возможность обращаться к нему через JMS, а можете сделать и то и другое, чтобы внешние системы могли пользоваться возможностями вашего роута, кто-то через вызовы web сервиса, а кто-то путем обмена сообщениями через JMS очереди.

Endpoints are also used so that routes can interact with each other. One route can send a message to the endpoint that another route reads, thus passing the message to it for processing. This allows you to create your own palette of routes that implement various functions that are in demand in different places of your application. For example, logging messages. Instead of performing the same set of logging actions each time, you can simply transfer message processing to a specially designed route.

Integration architecture

Our company uses a number of information systems. In order to organize an integration environment through which systems will exchange data, you first need to connect our systems to an integration platform. For this, an adapter is being developed for each system on ServiceMix, which will be responsible for exchanging data with the system and converting data formats.


For each system, one data exchange technology is selected between the system and ServiceMix. You can choose several, but this complicates the implementation, both on the system side and on the ServiceMix side. In the general case, the use of several different technologies is not justified, but it can technically be implemented. We mainly use RabbitMQ for exchanging with systems (we create sets of queues through which ServiceMix exchanges messages with integrated systems). But there are other cases in which the set of ready-made adapters that is part of ServiceMix really helps us. For example, in the case of our accounting system, we use stored procedures in the database. To call stored procedures, we use the MyBatis component, which allows you to map messages that go through ServiceMix to stored procedure parameters.

        {call esb.SetLogin (
       @UserId=#{UserId, mode=IN},
       @LoginId=#{LoginId, mode=IN}
        )}

Adapters are also responsible for converting the formats in which systems interact with the integration platform to the internal format. Someone prefers to exchange in JSON format, someone uses their own XML format, but inside the integration platform, the components exchange data in the internal canonical XML format. Now XML is losing popularity due to its heaviness and the emerged alternatives in the form of JSON, Protobuf, etc. But, in my opinion, XML is still convenient for solving integration problems. There are many useful technologies, such as XSLT and XPATH, that greatly simplify life, plus it is completely human-readable.

Message routing between components (adapters) is based on routing rules. Routes within one component of the integration platform interact through Camel's internal endpoints (direct, seda). Between each other, components exchange via RabbitMQ queues. This allows you to make the components of the integration platform independent. The fall of one component does not affect the performance of others. With a temporary increase in the load, messages will accumulate in the component queue without affecting the rest of the exchange.



Why such an architecture is better than direct interaction between systems on a point-to-point basis:

• Each system needs to support only one protocol and interaction format.
• Details of the exchange with other systems are hidden from the systems behind the integration platform. Systems interact only with the integration platform and may not even be aware of the existence of other systems, at least not worry about the features of interaction with them.
• Systems are loosely coupled, which gives additional flexibility, for example, if you need to replace one system with a new one.
• Tools for monitoring, testing, integration support are centralized on the side of the integration platform.

fault tolerance

Failover at the RabbitMQ level is achieved by creating a cluster and synchronizing queues. Each message that enters the queue on one of the RabbitMQ nodes is replicated to other nodes according to the synchronization policy (you can replicate to all nodes of the cluster, you can replicate to a certain number of nodes, you can not replicate at all). We use a three-node configuration with message replication for all nodes. This allows you to maintain the full efficiency of the cluster in the event of a fall of two of the three nodes. But you need to understand that this is also the most costly option in terms of processing time for each message and the occupied disk space. All three nodes are registered on the client to RabbitMQ, while the connection opens to one of the nodes, but if it falls, the client transparently switches to other nodes and continues to work.

ServiceMix fault tolerance is achieved through the use of a cluster of two ServiceMix nodes. At the same time, part of the bundles work in parallel, if this is permissible. For example, adapters that read one RabbitMQ queue, I can do it in parallel, only one of them will always receive one message, but at the same time messages will be evenly distributed between two nodes. In the event of a fall of one of the nodes, the second node takes on the entire load. Some bundles are active on only one node, but when it falls, bundles are activated on the second node. For example, if the adapter reads a shared network directory, then reading the same file at the same time can lead to duplication and collisions. This mode of operation is achieved through the use of shared locks based on Hazelcast. The bundle who first grabbed the lock

Hello world

In conclusion, I want to give a simple example of the Hello world application on Camel, which we will run on ServiceMix. Our application will simply write the phrase “Hello world” once in the ServiceMix log when the bundle starts (for example, when the bundle is deployed or ServiceMix is ​​restarted).

1. Download the ServiceMix distribution, unpack and run servicemix.sh (bat).
2. We create and configure a new maven project.
   In src / main / resources you need to create the META-INF directory in which to create the spring subdirectory.
   Because we need to build a bundle - edit pom.xml (add packaging and build instructions):
   
   

   4.0.0HelloWorldHelloWorld1.0.0-SNAPSHOTHelloWorldHelloWorldbundleorg.apache.felixmaven-bundle-plugin3.0.1true${project.artifactId}*${project.groupId};version=${project.version}

   
   No maven dependencies need to be added.
   
3. Configure Camel context and route.
   In the spring directory, you need to create the camel-context.xml file (the loader automatically searches for the Camel context description file in META-INF / spring and starts the routes). In the camel-context.xml file, put the following content (comments are given along the text):
   
   

   Hello world

   
   In order for our route to complete its task (logging the text “Hello world”), you need to send a message to it. In our case, the timer solves this problem., which, following the repeatCount = 1 instruction , will once send a message to the route input. Because the timer sends an empty message to the route input, we need to fill it with something - we put the text “Hello world” in the message body. At the end of the route, display the contents of the message body in the log.
   
4. Putting our project together : mvn package
5. Deployment assembled bundle.
   One way to deploy a bundle in ServiceMix is ​​to copy the jar file to the deploy directory. Copy the assembled jar to the ServiceMix / deploy directory .

Now look at the ServiceMix log: ServiceMix / data / log / servicemix.log . In addition to the information that we installed and launched the new bundle, “Hello world” should appear:

HelloWorldRoute | 43 - org.apache.camel.camel-core - 2.16.3 | Hello world

Try logging into the ssh console and viewing the Camel list of context-list contexts and the list of route-list routes. In the output of the commands, you will find our HelloWorldContext and HelloWorldRoute.

conclusions

In conclusion, I want to say that ServiceMix and Camel are excellent products for building integration solutions. The functionality is amazing, for almost any task you can find an elegant and simple solution. It can be seen that the products are very well thought out, the developers really worked hard. Our company ServiceMix has been used for 2 years, at the moment we have integrated about 16 of our information systems and have developed more than 150 integration services. There were still problems with some components, but there are always similar components from which you can choose or, in extreme cases, develop your own component. In general, the products work stably and reliably, personally my impression is the most positive. An undoubted advantage is the openness of the source code and the absence of the need to buy a license. Moreover, the products are absolutely not inferior to commercial counterparts. I also want to note that ServiceMix and Camel can be used not only for solving integration problems. For example, Karaf is perfect for deploying web applications into it. We use this opportunity to provide web interfaces to administrators for setting up an integration platform. Camel integrates seamlessly with various business components. For example, we use Drools (business rules engine) to assign incoming transactions to the appropriate portfolio according to certain rules (transaction enrichment with a portfolio), which are configured by users from the middle office. A very interesting product is Activiti (BPM engine), which can work in a Karaf container and integrate with Camel.