Modules instead of microservices

    The term module is taken from Modules vs. microservices . Also, the terms "microlith" or "monoservice" are sometimes used to describe something in between microservices and monoliths. But, despite the fact that the term "module" is already loaded with a well-known meaning, in my opinion it fits better than other options. Update: In a comment, lega used the term “embedded microservice” - it describes the essence of the approach better than “module”.


    Monolith and microservices are very different approaches, therefore, in any attempt to take the best from both, balance is crucial - what to take and what not. Otherwise, you get a monster like OSGi .


    I have been writing microservices since 2009, but I have not tried using modules instead of microservices in real projects yet - everything described below is my assumption about what the above balance should be, and it needs both theoretical criticism and practical verification.


    What is a module?


    A module is something like a microservice, only implemented within a single application, consisting of a group of such modules and a very small part, which is engaged in the initialization and launch of all these modules.


    Although formally such an application can be called a monolith, this approach has much more in common with microservices, therefore, it makes sense to compare it with the microservice approach. Like a microservice, each module:


    • It must have a carefully designed and documented API, and only it should be accessible to other code running in the same application (control of this restriction requires support at the language level and / or development tools).
    • It must be executed regardless of the code of other modules and be able to effectively interact with the API of other modules without using network protocols (this requires support at the language level: threads / goroutins / actors, channels / messages, etc.).
    • Must use its own data warehouse (if it is needed).
    • It can be in a separate repo or mono repo application.
    • It can be used by different applications.
    • It can provide access to its API over the network (only the module does not have to do this, and access may not be complete, as inside the application, but partial).

    Unlike microservices, the module:


    • Usually you can’t deploy independently (unless the language supports hot-swapping of part of the code in the running application).
    • Cannot be restarted regardless of the entire application.
    • You cannot scale independently of other modules.
    • It cannot interfere with access to "their" files from other modules if they decide to break the isolation between the modules. Neither language nor tools can prevent such a violation, so here you have to rely solely on the discipline of the developers themselves and the code review.
    • Perhaps it can use libraries shared with other modules without restrictions - but here I am not sure. The negative consequences of this will not be as significant as for conventional microservices, but still they will. Especially if the module will be launched in the future as a full-fledged microservice (and this, by and large, so far does not interfere - the modules are quite easily converted into separate microservices if necessary).

    Unlike regular libraries, the module:


    • It should not share any common data with the API code that uses it - all data must be transmitted through the API either as a copy, or if there is a lot of data and they need read-only access, as an immutable data structure (requires support on language level).
    • It does not provide functions that other modules can call - i.e. It does not execute any of its code in a foreign thread of execution (thread, goroutine, etc.), thus isolating even exceptions that may occur in the module code from the calling code.
      • The module has a public function for initializing and starting the module, which is called when the application starts, but it is not intended for (re) calling from other modules.
    • It has its own, completely isolated from other modules:
      • Configuration (passed to him when the application starts).
        • It can include logging settings common to all modules.
      • Data warehouse (if he needs it).
        • Support for versioning the schema of this data and migrations during updates is also the responsibility of the module, although starting data migrations of each module can be part of the application deployment process common to all modules.

    In most cases, the modules do not need any registry of services - instead of registering themselves and searching for other modules, they get the interfaces of other modules they need at startup, when the starting application calls the initialization function of each module (in the order determined by the dependencies between the modules). As a side effect, this will immediately detect cyclic dependencies between modules if they appear (and, if possible, change the architecture so as to get rid of them).


    Where are the modules needed


    There is a constant added complexity ( accidental complexity ), the same in each microservice (registration / discovery of services, connecting and reconnecting to them, authorization between services, (de) marshaling and encryption of traffic, use of loopback request breakers, implementation of request tracing, etc.). There is a similar constant added operational complexity (the need to automate testing and roll-out, implement detailed monitoring, aggregate logs, use service services to register and search for services, to store the configuration of services, for audit, etc.). You can put up with this because you can implement all this once, as the number of microservices grows, this complexity does not grow, and the advantages of microservices more than compensate for these costs.


    But there is a complexity that depends on the business logic of a particular application and increases as the application develops, which I would like to get rid of at least in those applications that do not require the ability to scale and high availability (or at least that part of the code of such applications that does not have obvious need to interact with external services):


    • The need to separate idempotent and non-idempotent requests.
    • For all requests on the client:
      • The need to use timeouts (which may be different for different requests).
      • The need to use more asynchrony to speed things up.
      • More situations of eventual consistency and the resulting difficulties.
        • Including the need to sometimes cache data from other services.
      • The need to repeat some queries:
        • Using a limit on the number or time of retries.
        • Using delays between repetitions.
    • Additionally, for non-idempotent client requests:
      • The need to realize the ability to identify duplicates when repeating queries.
    • Additionally for non-idempotent server requests:
      • The need to identify and process duplicate requests (differently, depending on the business logic of each request).
    • In some cases, the need to fulfill requests in the same order in
      which they were sent.

    The correct modular approach allows you to preserve many of the advantages of microservices (if you have the necessary support at the language level and / or development tools), but in addition to losing the unnecessary scalability and high availability features in this application, there are others:


    • A module crash causes the entire application to crash.
    • Build and test speeds are reduced.
      • Not sure build speed is a real problem - there are many ways to speed it up.
      • In principle, it is possible when testing the branch / PR to run only tests of the changed module, and to run the tests of the entire project only before the deployment - this should sufficiently level this problem.
    • Startup speed decreases (warming up caches, etc.).
      • Here the problem is not so much in the real slowdown, but in the fact that when a lot of services start up time, it’s "spread out" over several services that (re) start independently. However, the visible effect of a slower start does not cancel this fact.

    The modular approach also has new advantages:


    • The speed of interaction between service modules increases, which significantly reduces the need for asynchrony, eventual consistency, and caching.
    • As long as a module does not have an external (network) API, its API is much easier to change and deploy these changes atomically with a deployment of clients using this module (other modules of the same application).
    • Collaboration between modules is easier to test than a group of microservices.
    • It is easier to use a monorepo (if you want), although this is not necessary.
      • Monorepo simplifies refactoring and changing the API, at least as long as there is a guarantee that the module does not have a network interface and external clients.

    Summary


    In general, the approach looks quite tempting - we get the opportunity to write a monolithic application in the form of a bunch of really well-isolated parts (moreover, this will be controlled mainly by the language and / or tools, rather than the internal discipline) developed in the microservice style (in including by different teams in different repos), which with a “flick of the wrist” can turn into real microservices if there is a real need for it ... In the meantime, we can use messaging between the mod s within the application as a simple and very quick replacement of the RPC, eliminating the complexity of asynchrony, eventual consistency and handling of network errors.


    The required support for this approach is currently not available in all languages, but in some there are: the author of the article "Modules vs. microservices" wrote about modularity support in Java 9, Go has been supporting internal packages for a couple of years, judging by Erlang An article on the same topic, Dawn of the Microlith - Monoservices with Elixir, is doing well ... I'm not sure how much it is possible to provide real isolation of modules in scripting languages, but there are attempts: micromono on NodeJS, link lega to Python approach in comment lega ...


    If you have thoughts on the topic (or even better - experience of a real project on similar principles) or additional links to articles / projects on the topic - write in the comments, I will try to supplement them with the article.


    Also popular now: