xenvman: Flexible environments for testing microservices (and not only)

Hello!


I would like to tell you a little about the project I have been working on for the past six months. I do the project in my spare time, but the motivation for its creation came from observations made at the main work.


On the working draft, we use the microservice architecture, and one of the main problems that has manifested itself over time and with the increased number of these same services is testing. When a service depends on five to seven other services, plus some other database (or even a few) in addition, it is very inconvenient to test it in a “live” form, so to speak. It is necessary to be surrounded by mocks from all sides so tightly that the test itself cannot be seen. Well, or somehow organize the test environment, where all the dependencies could actually be running.


Actually to facilitate the second option, I just sat down to write xenvman . In a nutshell, it is something like a hybrid of docker-compose and test containers , only without binding to Java (or any other language) and with the ability to dynamically create and configure environments via the HTTP API.


xenvman written on Go and implemented as a simple HTTP server, which allows you to use all the available functionality from any language that can speak this protocol.


The main thing that xenvman can do is:


  • Flexible description of the contents of the environment using simple JavaScript scripts
  • Create images on the fly
  • Create the required number of containers and combine them into a single isolated network
  • Forwarding the internal ports of the environment to the outside, so that tests can reach the necessary services even from other hosts
  • Dynamically change the composition of the environment (stop, start and add new containers) on the go, without stopping the working environment.

Environments


The main protagonist in xenvman is the environment. This is such an isolated bubble in which all the necessary dependencies (packed in Docker containers) of your service are launched.



The figure above shows the xenvman server and active environments where various services and databases are running. Each environment was created directly from the integration test code, and will be deleted upon completion.


Templates


What is directly included in the environment is determined by the templates (templates), which are small scripts in JS. xenvman has a built-in interpreter for this language, and when it receives a request to create a new environment, it simply executes the specified templates, each of which adds one or more containers to the list for execution.


JavaScript was chosen to allow dynamically change / add templates without having to rebuild the server. In addition, as a rule, only basic features and data types of the language are used in the templates (good old ES5, no DOM, React and other magic), so working with templates should not cause any particular difficulties even for those who do not know JS.


Templates are parameterizable, that is, we can completely control the template logic by passing certain parameters in our HTTP request.


Creating images on the fly


One of the most convenient features of xenvman, in my opinion, is the creation of Docker images right along with the configuration of the environment. Why it may be necessary?
Well, for example, we have on a project to get an image of a service, we need to commit changes into a separate branch, push and wait while Gitlab CI collects and floods the image.
If only one service has changed, then it can take 3-5 minutes.


And if we are actively sawing new features in our service, or trying to understand why it does not work, adding the good old ones fmt.Printfback and forth, or changing the code sometime, even a delay of 5 minutes will be great to put out the performance (our code writers). Instead, we can simply add all the necessary debugging to the code, compile it locally, and then simply attach the finished binary to the HTTP request.


Having received such good, the template will take this binar and on the move will create a temporary image from which we can already launch the container as if nothing had happened.


On our project, in the main template for services, for example, we check if the binary is present in the parameters, and if so, then collect the image on the go, otherwise just download the latestversion of the devbranch. The following code for creating containers is identical for both options.


Small example


For clarity, let's look at the micro-sample.


Let's say we write some kind of miracle server (let's call it - wut), which needs a database to store everything there. Well, as a base, we chose MongoDB. So for full testing, we need a working Mongo server. You can, of course, install and run it locally, but for simplicity and clarity of the example, we assume that for some reason this is difficult to do (with other, more complex configurations in real systems, this will be more like the truth).


So we will try to use xenvman to create an environment with Mongo running and our wutserver.


First we need to create a base directory in which all templates will be stored:


$ mkdir xenv-templates && cd xenv-templates


Next, create two templates, one for Mongo, the other for our server:


$ touch mongo.tpl.js wut.tpl.js


mongo.tpl.js


Open mongo.tpl.jsand write the following there:


functionexecute(tpl, params) {
  var img = tpl.FetchImage(fmt("mongo:%s", params.tag));
  var cont = img.NewContainer("mongo");
  cont.SetLabel("mongo", "true");
  cont.SetPorts(27017);
  cont.AddReadinessCheck("net", {
    "protocol": "tcp",
    "address": '{{.ExternalAddress}}:{{.Self.ExposedPort 27017}}'
  });
}

The execute () function with two parameters must be present in the template .
The first is the tpl instance of the object through which the environment is configured. The second argument (params) is just a JSON object, with which we will parameterize our template.


In line 


var img = tpl.FetchImage(fmt("mongo:%s", params.tag));

we ask xenvman to download the docker image mongo:<tag>, where <tag>is the version of the image we want to use. In principle, this line is equivalent to the command docker pull mongo:<tag>, with the only difference that all the functions of the tplobject are essentially declarative, that is, the actual image will be downloaded only after xenvman executes all the templates specified in the environment configuration.


Once we have an image, we can create a container:


var cont = img.NewContainer("mongo");

Again, the container will not be instantly created in this place, we simply declare the intention to create it, so to speak.


Next, we label our container:


cont.SetLabel("mongo", "true");

Labels are used so that containers can find each other in the environment, for example, to enter the IP address or host name in the configuration file.


Now we need to hang the inner Mongo port (27017) out. This is easily done like this:


 cont.SetPorts(27017);

Before xenvman reports on the successful creation of the environment, it would be great to make sure that all services are not just running, but already ready to accept requests. Xenvman has readiness checks for this .
Add one for our mongo container:


 cont.AddReadinessCheck("net", {
    "protocol": "tcp",
    "address": '{{.ExternalAddress}}:{{.Self.ExposedPort 27017}}'
  });

As we can see, here in the address bar there are stubs into which the necessary values ​​will be dynamically substituted just before launching the containers.


The {{.ExternalAddress}}external address of the host on which xenvman is running {{.Self.ExposedPort 27017}}will be substituted instead, and the external port that was forwarded to the internal 27017 will be substituted instead .


Read more about interpolation here .


As a result of all this, we will be able to connect to Mongo, running in the environment, right outside, for example, from the host on which we run our test.


wut.tpl.js


So-c, having understood Monga, we will write another template for our wutserver.
Since we want to build the image on the go, the pattern will be slightly different:


functionexecute(tpl, params) {
  var img = tpl.BuildImage("wut-image");
  img.CopyDataToWorkspace("Dockerfile");
  // Extract server binaryvar bin = type.FromBase64("binary", params.binary);
  img.AddFileToWorkspace("wut", bin, 0755);
  // Create containervar cont = img.NewContainer("wut");
  cont.MountData("config.toml", "/config.toml", {"interpolate": true});
  cont.SetPorts(params.port);
  cont.AddReadinessCheck("http", {
    "url": fmt('http://{{.ExternalAddress}}:{{.Self.ExposedPort %v}}/', params.port),
    "codes": [200]
  });
}

Since here we collect the image, we use BuildImage()instead FetchImage():


var img = tpl.BuildImage("wut-image");

In order to build an image, we will need several files:
Dockerfile is the actual instruction how to build the image.
Config.toml is the configuration file for our wutserver


Using the method, img.CopyDataToWorkspace("Dockerfile");we copy the Dockerfile from the template data directory to a temporary working directory .


The data directory is the directory in which we can store all the files our template needs in the work.


In the temporary working directory, we copy the files (using img.CopyDataToWorkspace ()), which will fall into the image.


Then follows this:


// Extract server binaryvar bin = type.FromBase64("binary", params.binary);
  img.AddFileToWorkspace("wut", bin, 0755);

We pass our server's binary directly in the parameters, encoded (base64). And in the template, we simply decode it, and save the resulting string to the working directory as a file under the name wut.


Then we create a container and mount the configuration file into it:


var cont = img.NewContainer("wut");
 cont.MountData("config.toml", "/config.toml", {"interpolate": true});

The call MountData()means that the file config.tomlfrom the data directory will be mounted inside the container under the name /config.toml. The flag interpolateindicates to the xenvman server that, before mounting it in the file, all the stubs existing there should be replaced.


Here is what the config might look like:


{{with .ContainerWithLabel "mongo" "" -}}
mongo = "{{.Hostname}}/wut"
{{- end}}

Here we are looking for a container with a label mongo, and substitute the name of its host, whatever it is in this environment.


After substitution, the file may look like:


mongo = “mongo.0.mongo.xenv/wut”

Then we post the port again and start the readiness check, this time HTTP:


cont.SetPorts(params.port);
 cont.AddReadinessCheck("http", {
    "url": fmt('http://{{.ExternalAddress}}:{{.Self.ExposedPort %v}}/', params.port),
    "codes": [200]
  });

On this, our templates are ready, and we can use them in the code of integration tests:


import"github.com/syhpoon/xenvman/pkg/client"import"github.com/syhpoon/xenvman/pkg/def"// Создаём xenvman клиент
cl := client.New(client.Params{})
// Требуем создать для нас окружение
env := cl.MustCreateEnv(&def.InputEnv{
    Name:        "wut-test",
    Description: "Testing Wut",
    // Указываем, какие шаблоны добавить в окружение
    Templates: []*def.Tpl{
        {
            Tpl: "wut",
            Parameters: def.TplParams{
                "binary": client.FileToBase64("wut"),
                "port":   5555,
            },
        },
        {
            Tpl: "mongo",
            Parameters: def.TplParams{"tag": “latest”},
         },
    },
})
// Завершить окружение после окончания тестаdefer env.Terminate()
// Получаем данные по нашему wut контейнеру
wutCont, err := env.GetContainer("wut", 0, "wut")
require.Nil(t, err)
// Тоже самое для монго контейнера
mongoCont, err := env.GetContainer("mongo", 0, "mongo")
require.Nil(t, err)
// Теперь формируем адреса
wutUrl := fmt.Sprintf("http://%s:%d/v1/wut/", env.ExternalAddress, wutCont.Ports[“5555”])
mongoUrl := fmt.Sprintf("%s:%d/wut", env.ExternalAddress, mongoCont.Ports["27017"])
// Всё! Теперь мы можем использовать эти адреса, что подключиться к данным сервисам из нашего теста и делать с ними, что захочется

It may seem that writing templates will take too much time.
However, with the right design, this is a one-time task, and then the same templates can be re-used again and again (and even for different languages!) Simply by fine-tuning them by passing certain parameters. As you can see in the example above, the test code itself is very simple, due to the fact that all the husks for setting up the environment we have taken out in the templates.


In this small example, not all the features of xenvman are shown, a more detailed step-by-step tutorial is available here.


Customers


At the moment there are clients for two languages:


Go
python


But adding new ones is not difficult, since the provided API is very, very simple.


Web interface


In version 2.0.0, a simple web interface was added, with which you can manage environments and view available templates.





How is xenvman different from docker-compose?


Of course, there are many similarities, but xenvman seems to me a bit more flexible and dynamic approach, compared to the static configuration in the file.
Here are the main distinguishing features, in my opinion:


  • Absolutely everything is managed through the HTTP API, therefore we can create environments from code of any language that understands HTTP.
  • Since xenvman can be run on a different host, we can use all its capabilities even from a host on which docker is not installed.
  • The ability to dynamically create images on the fly
  • The ability to change the composition of the environment (add / stop containers) in the process of its work
  • Reducing the boilerplate code, improving the composition and the possibility of reuse of the configuration code due to the use of parameterizable patterns

Links


Github project page
Detailed step-by-step example, in English.


Conclusion


That's all. In the near future I plan to add the ability to
call templates from templates and thereby allow them to be combined with greater efficiency.


I will try to answer any questions, and I will be glad if this project turns out to be useful to anyone else.

Tags:

Also popular now: