Node.js and cote: simple and convenient development of microservices

Many people think that microservices are very difficult. In fact, with the right approach, this is not at all the case.

Microservices today are very popular, and true adherents of this architecture almost bow to everything that the "microservice" is written on. However, if you discard fanaticism, such an approach to software development is a worthy step forward, microservices can forever change the way the server parts of applications are created. There is a lot of informational noise around microservices, so it’s worth highlighting the truly important properties of this architecture and working to simplify its implementation and use where it is really needed.



If you are wary of microservices, or feel that you are confused about this topic, know that you are not alone. From an architectural point of view, microservices are not so simple. The matter is compounded by the ecosystem around them. Instead of directly solving the complex tasks associated with working with microservices, those who do this try to get away from direct solutions, simplifying one and complicating the other. As a result, this leads to the fact that you have to maintain a wide range of technologies and deal with errors that cannot be debugged precisely because of the complexity of the entire structure. It’s not easy to just check the application’s performance The time has come to rectify this situation.

Here we will walk you through the development of a microservice-based web application step by step. This example may well be your first such project, and if you understand the development for Node.js, it will take you a few minutes to get everything working.

Before we get down to business, I would like to clarify that I am the author of the library for Node.js cote , which simplifies and accelerates the development of applications based on microservices.

An example of a microservice-based application

There are many ways to demonstrate the implementation of microservice architecture, and, in fact, a thoroughly worked out example is all that an experienced developer needs. However, this material is intended for programmers with different levels of training, including beginners, so a simple but complete example was chosen for demonstration: an application for converting currencies. We will create a system of four services, the joint work of which will allow us to organize the conversion in the conditions of fluctuating exchange rates.


Overview of services. Each of them can be independently scaled. Arrows indicate the direction of data flows.

We will create a system consisting of the following parts.

1. Currency converter client (conversion client) - he can request the conversion of a certain amount of money in one currency into another currency.
   
   
2. Currency converter service (conversion service) - he knows the exchange rates, he can respond to customer requests. In addition, this microservice may receive updates regarding currency translation rates.
   
   
3. Arbitration service (arbitration service) - it maintains up-to-date information on exchange rates, and, when changed, publishes updates.
   
   
4. The arbitration admin tool (arbitration admin) is a service that you can implement yourself, as your homework. He is needed to manage the arbitration service, allowing you to inform him of the need to update any exchange rate.

Before we get down to business, let's talk a bit about the history of microservices.

Subjective look at the history of microservices

Let's take a look at the past. Ten years ago, when microservices had not yet appeared, service-oriented architecture (SOA) was in high esteem. There were hundreds of solutions for building service-oriented applications, and a serious consulting business was built around all this.

Let's move seven years closer to our time, here we will already meet with microservices. Although at a fundamental level, they are very different from SOA, those who used to build service-oriented applications have switched to a new technology. Everything would be fine, but behind these developers there was baggage from the past, in which there is a lot of superfluous.

Overnight, existing development approaches were moved to a new environment. Although the basic premise underlying microservices was different from SOA, these first solutions, based on old technologies, were presented to the public as "the only true ones." As a result, among other things, we had AMQP, HTTP, or even, God forbid, SOAP. These were nginx, zookeeper, etcd, consul and several other solutions that tried to give the appearance of the tools vital for creating microservices.

The problem with all this was that these technologies were originally created to solve completely different problems. They were, so to speak, “soldered” to microservices. However, after the developers used some of these tools to build their own services, they, at best, left a feeling of temporary improvised means. It was like hammering nails with pliers. When there is nothing else at hand - you can hammer a nail, but is it not better to look for a normal hammer?

It was always obvious that the barrier to entry into the field of microservices is too high, although the transition to them promised economic benefits. However, outside the window of 2017, we all deserve better than the old technologies that help us solve new problems. It's time to say that in order to study, create and use microservices, in order to achieve their scaling in solving real problems, you only need Node.js.

Now let's talk about the architecture of modern microservices.

Five Microservice Requirements

Here are the requirements that modern systems based on microservices must meet.

1. Automatic system configuration . Any microservice-based system is likely to include hundreds of services. Manually configuring IP addresses, ports, and API capabilities is an almost impossible task.
   
   
2. High system redundancy . In the above system, microservice failures are quite possible, therefore, it is very important to have a failover mechanism for reserve capacities, the creation of which should not be particularly troublesome and costly.
   
   
3. System fault tolerance . The system must withstand and adequately handle abnormal events, such as network failures, errors during message processing, timeouts, and so on. Even if some services ceased to function, all others not related to them should work.
   
   
4. Self-healing system . Failures and various kinds of errors must be treated as completely expected phenomena. The implementation of the system should include the automatic restoration of any functionality or service lost due to a failure.
   
   
5. Auto discovery of services . Existing services should automatically identify new services introduced into the system, begin interaction with them without manual intervention or downtime.
   

If your architecture meets these requirements and if you split the execution of most of your API requests into several independent services, then what you did is microservices.

These are not microservices.

I would like to emphasize once again one important thing: the architecture of microservices is not tied to any particular technology. Say, good old work queues and consumer flows are not microservices. Mail daemons, notification mechanisms, any auxiliary services that only consume events and do nothing to process user requests are not microservices.

Are some administrative applications and client programs different? This separation also does not mean the use of microservice architecture. Server daemons that can receive and process HTTP requests ... And these are not microservices. But what if we have a cluster of servers, the computers included in which are named after the moons of Jupiter? Does the administrator need to intervene in their work? If so, then there are no microservices.

It is clear that in the wake of the popularity of the new technology, it is tempting to call everything “microservice architectures” in a row. However, this only creates informational noise and does not allow other people to understand what microservices are, and therefore to use them in their designs.

Microservices have always strived for minimalism. “Heavy” technologies, which are said to be the only ones that make microservices possible, are the exact opposite.

Cote library and microservices

Microservices built on the basis of the cote library are automatically configured. The library uses broadcast or multicast IP broadcasts; as a result, daemons on the same network find each other and automatically exchange information necessary for setting up the connection. In this regard, cote meets the requirements of No. 1 and No. 5: "Automatic system configuration" and "Automatic discovery of services."

Using cote, you can very efficiently, with low resources, create multiple copies of services, while request processing scales automatically. This means that the library also meets the requirement No. 2: "A high level of system redundancy."

If there are no services in the system to satisfy a certain request, cote caches it and waits until the required service is available. Since services are usually independent, such an organization of the system means its fault tolerance, and this corresponds to the requirement No. 3: "Fault tolerance of the system."

The remaining requirement # 4: “Self-healing the system”, is implemented through the use of Docker, making it possible to restart services in case of failure. Since the system automatically detects services, even if Docker decides to deploy the fallen service on a new machine, all remaining services will find it and will interact with it.

Thus, cote gives the developer complete freedom in terms of infrastructure, taking care of fulfilling the basic requirements for systems based on microservices.

With cote, you can focus on the most important aspects of application development, leaving the library with complementary tasks.

I suppose I spoke clearly and understandably about what real microservices are. Now let's talk about how to implement all this.

▍Install cote

We will create a system based on microservices in the Node.js environment using the cote library , the main features of which we examined above. It is available as an npm package for Node.js.

Install cote:

npm install cote

▍First acquaintance with cote

Do you want to integrate cote with an existing web application, for example, based on express.js, as shown here , are you going to rewrite some part of the monolithic application, or decided to transfer several existing microservices to cote, the work will be to create several instances cote components and use them according to your needs. Among these components, for example - Responder , Requester , Publisher , Subscriber , you will learn more about them below. These components are designed to interact with each other, allowing you to implement the most common application development scenarios.

A simple application, or a very small microservice, can very well be built so that one Node.js process will have to implement one cote component. However, more complex projects require closer communication and collaboration of many components and microservices. The cote library also supports such scripts.

Let's start with the client whose file we’ll name conversion-client.js. Its role is to request a currency conversion operation and, upon receipt of an answer, to carry out some actions.

Implementation of request and response mechanism

The most common component interaction scenario implemented in many applications is a request-response cycle. Typically, a microservice requests a task from another microservice, or executes a request to it and receives a response from it. We implement a similar interaction scheme using cote.

To get started, in the file conversion-client.js, connect cote.

const cote = require('cote');

Nothing special happens here - a normal library call.

▍Create the Requester component

Let's start with the component Requesterthat will request the implementation of the currency conversion operation. And Requester, and other components, these are constructor functions in the cote module, so they are created using the keyword new.

const requester = new cote.Requester({ name: 'currency conversion requester' });

As the first argument, cote component constructors take an object that, at a minimum, should contain a property namethat is the name of the component used to identify it. The name is used mainly as an identifier for monitoring components, it is very useful when reading logs, since, by default, each component logs the names of other components it discovered.

The components created by the constructor Requesterare designed to send requests, it is assumed that they will be used together with the objects that the constructor creates Responderthat respond to requests. If no such components are found, the component Requesterwill accumulate requests in the queue until it appearsResponderto which you can send these requests. If Responderthere are several components , he Requesterwill access them using the circular polling algorithm, balancing the load on them.

We will create a request of the type convertthat is designed to convert a certain amount, expressed in US dollars, into euros.

const request = { type: 'convert', from: 'usd', to: 'eur', amount: 100 };
requester.send(request, (res) => {
    console.log(res);
});

Recall that so far we have been working in a file conversion-client.js. Here, just in case, its full code.

const cote = require('cote');
const requester = new cote.Requester({ name: 'currency conversion requester' });
const request = { type: 'convert', from: 'usd', to: 'eur', amount: 100 };
requester.send(request, (res) => {
    console.log(res);
});

In order to execute this file, use this command:

node conversion-client.js

Now the execution of the request does not lead to anything useful, there are not even logs in the console, since our system does not yet have a component that can respond to the request and return something in response.

Let this node.js process be executed, but for now we Responderwill deal with a component that can respond to requests.

▍Creating a Responder component

First, as last time, connect cote and create an instance of the object Responderusing the keyword new.

const cote = require('cote');
const responder = new cote.Responder({ name: 'currency conversion responder' });

This part of our system will be represented by a file conversion-service.js. Each Responder, among other things, is an instance of an object EventEmitter2. The answer to a request, say, of a type convertis the same as listening to an event convertand processing it using a function that takes two parameters — a request and a callback function. The request parameter contains information about one request, in general, this is the same object requestthat was sent by the component Requesterdiscussed above. The second parameter is a callback function, which is called with the transmission of what should be sent in response to it.

Here's what a simple implementation of the above mechanism looks like.

const rates = { usd_eur: 0.91, eur_usd: 1.10 };
responder.on('convert', (req, cb) => { // в идеале, тут хорошо бы привести в порядок входные данные
    cb(req.amount * rates[`${req.from}_${req.to}`]);
});

Here is the complete file code conversion-service.js.

const cote = require('cote');
const responder = new cote.Responder({ name: 'currency conversion responder' });
const rates = { usd_eur: 0.91, eur_usd: 1.10 };
responder.on('convert', (req, cb) => {
    cb(req.amount * rates[`${req.from}_${req.to}`]);
});

Save the file, and, in the new terminal, execute it using node:

node conversion-service.js

Immediately after starting the service, you will see how the first request from conversion-client.js. Information about this will go to the log. As a matter of fact, knowing everything that we just talked about, you can already start creating your own microservices.

Please note that we did not configure IP addresses, ports, host names, or anything else. And also - congratulations! You have just created your first set of microservices.

Now, in different terminals, you can start many copies of each service and make sure that all this works fine. Stop several conversion services, restart them, and you will see that the system we created meets the requirements for modern microservices. If you want to scale the project, deploy it on several servers or in several data centers, you can either use your own solution, or use the capabilities of Docker, which allows you to solve many infrastructure management tasks.

We will further develop our currency converter.

Track system changes using the publisher-subscriber mechanism

One of the advantages of systems based on microservices is that it is very easy to implement mechanisms that previously required serious investments in infrastructure. Among the tasks solved by using such mechanisms is updating management and tracking changes in the system. Previously, this required, at a minimum, forked queuing infrastructure. Such things are not easy to scale, they are difficult to manage, this was one of the limiting factors in the development of such systems.

The cote library solves such problems in a completely natural and intuitive way.

Suppose we need an arbitrage service that sets exchange rates, and if changes have occurred, it notifies all instances of the converter services that they need to use the new exchange rates.

The most important thing here is the notification of all instances of the converter services. In a highly accessible application based on microservices, you can expect to have several identical services between which the load is distributed. When exchange rates are updated, all of these services should be informed of the changes. If the application had only one service converter, this would be easy to implement using the request-response mechanism. But since we strive for a scalable architecture, we don’t want to limit ourselves to the number of simultaneously working services, we need a mechanism that would notify each of these services, and it is necessary that they all receive similar notifications at the same time. In cote, this is achievable through a mechanism of publishers and subscribers.

Of course, the arbitration service itself is not independent, it must be managed through some API so that it can receive information about new courses through special requests. As a result, for example, the administrator will be able to enter into the system information about new courses, which, as a result, will reach the converter services. In order to achieve this, the arbitration service must include two components. One of them is the component Responderresponsible for implementing the API for updating courses from an external source, and the second is the component Publisherthat publishes updates by notifying the converter services. In addition to this, the converter services themselves should include a component Subscriberthat will allow them to subscribe to course updates. Let's see how to do it all.

▍Creation of arbitration service

We develop an arbitration service. To get started, connect cote and create an object Responderfor the API. And now - a small but very important detail regarding the creation of microservices using cote. Since the objects are Requesterautomatically configured, each of them will be connected to each object Responderthat it will be able to detect, regardless of the types of requests this one Respondercan respond to. This means that everyone Respondermust answer exactly the same set of requests, since the objects Requesterwill distribute the load between all the objects Responderto which they are connected, regardless of their capabilities, that is - not paying attention to whether they can Serve requests of a certain type.

In this case, we are going to create a componentResponderfor a set of requests that differs from that which microservices existing in the system can already process. This means that we need the new one to be Responderdifferent from regular services that handle type requests convert. In cote, this can be done by specifying the component key. Keys are the easiest way to regulate the interaction of services. Here's how to create a component Responderfor an arbitration service.

const cote = require('cote');
const responder = new cote.Responder({ name: 'arbitration API', key: 'arbitration' });

Now we need a mechanism to track exchange rates in the system. Let's say they are stored in a local variable in the scope of the module. It may well be the database accessed by the service, but in order not to complicate the example, let it be a local variable.

const rates = {};

Now the component Responderwill have to respond to requests of the type update rate, allowing the administrator to update exchange rates using the utility application. At the moment, the integration of our system with an auxiliary application, something like an administrator’s office, is not important, however, here is an example of how such an application can interact with components Responderimplemented using cote running on a server. The arbitration service should have a component Responderthat can accept information about new exchange rates.

responder.on('update rate', (req, cb) => {
    rates[req.currencies] = req.rate; // { currencies: 'usd_eur', rate: 0.91 }
    cb('OK!');
});

As an exercise, you can create a component Requesterso that, for example, it periodically makes queries of the type update rate, while changing courses. Name the file with this mechanism arbitration-admin.jsand implement in it something like a timer based on setIntervalwhich each time gives different exchange rates for the arbitration service.

▍Creating the Publisher component

Now we have a mechanism for updating exchange rates, but the rest of the system is not aware of this. In particular, we are talking about currency conversion services. In order to inform them of changes in rates, it is necessary, in the arbitration service, to use the component Publisher.

const publisher = new cote.Publisher({ name: 'arbitration publisher' });

Now, when updating courses, you need to take advantage of the capabilities of this component. As a result, the type request handler update ratewill need to be edited as shown below.

responder.on('update rate', (req, cb) => {
    rates[req.currencies] = req.rate;
    cb('OK!');
    publisher.publish('update rate', req);
});

Now the work on the arbitration service is completed, below, as usual, its full code is shown, which should be in the file arbitration-service.js.

const cote = require('cote');
const responder = new cote.Responder({ name: 'arbitration API', key: 'arbitration' });
const publisher = new cote.Publisher({ name: 'arbitration publisher' });
const rates = {};
responder.on('update rate', (req, cb) => {
    rates[req.currencies] = req.rate;
    cb('OK!');
    publisher.publish('update rate', req);
});

Since there are no subscribers to exchange rate update events in the system, converter services will not know that exchange rates have changed. In order for the publisher-subscriber mechanism to work, you need to return to the file you already know conversion-service.jsand add a component to it Subscriber.

▍Create a Subscriber component

The procedure for working with type components Subscriberis no different from other components. Create, in the file conversion-service.js, a new instance of the corresponding object.

const subscriber = new cote.Subscriber({ name: 'arbitration subscriber' });

Objects of the type Subscriberextend the functionality of objects EventEmitter2, and although these services can run on computers located on different continents, any new data coming from the publisher will ultimately be accepted by the subscriber as an event that needs to be processed.

Add to the conversion-service.jsfollowing code, which will allow you to listen to updates published by the arbitration service.

subscriber.on('update rate', (update) => {
    rates[update.currencies] = update.rate;
});

That's all. After all our microservices have been launched, the conversion services will be synchronized with the arbitration service, receiving messages they publish. Conversion requests received after updating the courses will be carried out taking into account the latest data.

We have just created three services, the joint work of which allowed us to implement a currency conversion system based on a microservice architecture. Here is the repository on GitHub, in which you can find all three microservices that we worked here, and a service for automatically updating exchange rates. If you want, clone the repository and experiment with its contents.

What's next?

If you are interested in the topic of developing microservices using cote, take a look at the library repository on GitHub, join our Slack community. The project needs active participants, therefore, if you want to contribute to the matter of turning microservices into a widespread, accessible tool for everyone, let us know.

Here is another repository where you can find an extended example, which is an implementation of a simple cote-based e-commerce application. In this example, in particular, the following is available.

• Administrative application with real-time update function for managing the product catalog and displaying sales information using the RESTful API (express.js).
  
  
• Страничка для пользователей-клиентов, с помощью которой они могут покупать товары. Она тоже обновляется в режиме реального времени, здесь используется технология WebSockets (socket.io).
  
  
• Микросервис для обслуживания нужд пользователей, выполняющий CRUD-операции.
  
• Микросервис для работы с продуктами, опять же, реализующий CRUD-операции.
  
• Микросервис, позволяющим клиентам совершать покупки.
  
• Платёжный микросервис, который занимается обработкой финансовых транзакций, возникающих как следствия покупок товаров.
  
• Конфигурация Docker Compose для локального запуска системы.
  
• Облачная конфигурация Docker для запуска системы в Docker Cloud.
  

If you want to experiment with cote in Docker or in Docker Cloud, here is a webinar entry . Here you will find a step-by-step guide, which, from scratch, demonstrates the creation of a working system based on microservices that supports scaling and continuous integration.

Summary

In this article, we, very briefly, examined the topic of developing applications based on microservices. This is an introduction to the architecture of microservices, a general overview of the development methodology. However, although the application that we created here consists of only a few lines of code, the same approach can be used on much larger projects. It should be noted that we considered the cote library only in general terms.

I believe that we are on the verge of very interesting times when you can observe a change in approaches to software development. I strongly support the simplification path that the web development industry follows. The cote library is one of the steps along this path, and, in fact, this is only the beginning. I hope this material helped those who thought about microservices, but did not dare to implement them in their projects.

Dear readers! Do you use microservices in your developments?