Rally: Testing OpenStack with Tempest Made Easy (Easier)
Author: Andrey Kurilin
In the modern world, when building cloud services, it is not possible to limit itself to installing the platform only, you have to upgrade components, integrate new hardware, scale and make many changes to the settings. But how can one remain confident that after all these changes, your cloud will continue to work properly and serve the customers for the benefit of the empire?
At this stage, Tempest (the official OpenStack component for functional testing) is used for these tasks. Tempest tests are automatically run for each new change in almost all OpenStack projects, which reduces the likelihood of applying incorrect changes (broken, curves, in other words, changes that can break anything in your infrastructure).
However, when using Tempest, users often encounter some inconvenience. Using Tempest reminds me of studying Assembler at a university. Writing and executing assembler instructions directly caused a lot of inconvenience, even if you couldn’t fix your mistakes without pain and suffering due to possible memory shifts. When we were offered the opportunity to use Turbo Pascal as an environment for launching assembler instructions, we were immensely happy; the process of studying and developing applications in Assembler was greatly simplified. I feel the same when, instead of tormenting myself with the Tempest configuration, I can test my cloud with a couple of simple commands.
In most cases, using Tempest is difficult and time consuming. In addition, Tempest does not support a number of functionalities for load testing, such as load simulation and presentation of results in graphical form. Rally
can solve these problems - the OpenStack project, which is capable of many things, in particular: creating environments for verification, testing performance, as well as OpenStack benchmarking (including using Tempest).
If we talk about Rally + Tempest, then Rally automates the process of installing, configuring and running Tempest tests. Unlike Tempest, which must be installed and configured separately on each cluster, once installed, the Rally service can check a huge number of clouds - you just need to add information about your clusters in Rally, and you can easily switch between them later. The Rally benchmarking engine is able to automatically run tests, simulating user load. The results of both verification and benchmarks are stored in the Rally database. You can view them immediately, or after a certain period, to compare how changes in the configuration of your cloud affect its operation.
Next, we will look at how to use Rally and Tempest, but before moving on to the most interesting, we list the main reasons for integrating Rally with Tempest:
1. Most OpenStack projects use CI / CD systems based on Tempest to automatically test new changes. Based on this, if there is an opportunity to make life easier for developers, namely to allow them to use Tempest more actively and conveniently, then we must do it!
2. Load testing using Tempest tests:
1. Many tests have already been developed within the framework of Tempest, so there is no need to duplicate them in Rally. The less code duplication, the better.
2. Verification should not be limited to no-load testing. Rally can simulate the load for Tempest. In other words, the benchmarking mechanism in Rally can run Tempest tests with a different number of (simulated) active users.
3. Debbuging Tempest and OpenStack. When a race condition is detected in gates, developers will be able to run the same test (or test suite) under load in their environment, which will make it easier to identify the source of this kind of error.
You can talk a lot about the cool and promising features of Rally, but you will never know how good Rally is until you test it in action. So, probably you are ready to move from the "lyrics" to practice. Then let's get started!
Not surprisingly, to use Rally, you must first download it ( github repository ), and then run the install_rally.sh installation script.
After a successful installation, you should add a cloud to Rally, which will be tested. This operation can be performed in two ways. One of the options for initializing a cluster in Rally is to use the OpenRC of your cloud. For example, like this: The
second method is to describe the cloud in the json file , which should be passed to the rally deployment create using the –file argument: By executing the
rally deployment list command, you can see a list of all initialized deployments.
Using the `rally deployment check` you can check the correctness of the added deployment, as well as see what services are available in your cloud from Rally:
As mentioned earlier, Rally can simultaneously work with a huge number of clouds, therefore, for each cloud custom Tempest. To reduce the time it takes to clone a Tempest repository, Rally clones it once and then uses the local repository, customizing it for each cloud.
Tempest preparation (cloning the repository and installing a virtual environment with all the dependencies) is done with the rally-manage tempest install command.
The start of automatic cloud verification (in other words, cloud verification) is implemented using the rally verify start command; it requires only one argument: the name of the test suite. If this argument is not specified, smoke tests will be performed (by default). The result of this command is similar to the result issued by Tempest.
Valid names for test suites are: full, smoke, baremetal, compute, data_processing, identity, image, network, object_storage, orchestration, telemetry and volume.
The results of all verification are stored in the Rally database, so you can view and compare them at any time. Lists of running checks will be displayed on the screen using the `rally verify list` command.
You can view the results of a specific verification with two commands: rally verify show and rally verify detailed. The latter complements the output of the first command with tracebacks of tests that failed.
In addition to installing, configuring, and running Tempest, Rally can model user workloads using Tempest tests. The results of stress testing are also saved and can subsequently be displayed.
Run the benchmark and simulate the load (from 2 to 10 users) for the following test scenario: tempest.api.identity.admin.v3.test_domains.DomainsTestXML.test_create_update_delete_domain.
After the test is completed, using the rally task plot2html command, we generate an HTML file with the results. Information and examples of how the input data are correctly set in the JSON / YAML files used to define this task can be found in the Rally repository (in the rally / doc / samples / tasks directory) or on the OpenStack wiki page .
When using clouds in production, you have to deal with constant changes. The question arises: how to check each change? Tempest is a good but cumbersome solution. Rally, among other things, can simplify its use.
Rally provides a mechanism for stress testing OpenStack, and now, thanks to integration with Tempest, the library of test scripts has become huge. Will your fix speed up any operations? Do you have the same fast cloud as your competitors? Why not figure it out with Rally ?!
Original article in English .
In the modern world, when building cloud services, it is not possible to limit itself to installing the platform only, you have to upgrade components, integrate new hardware, scale and make many changes to the settings. But how can one remain confident that after all these changes, your cloud will continue to work properly and serve the customers for the benefit of the empire?
At this stage, Tempest (the official OpenStack component for functional testing) is used for these tasks. Tempest tests are automatically run for each new change in almost all OpenStack projects, which reduces the likelihood of applying incorrect changes (broken, curves, in other words, changes that can break anything in your infrastructure).
However, when using Tempest, users often encounter some inconvenience. Using Tempest reminds me of studying Assembler at a university. Writing and executing assembler instructions directly caused a lot of inconvenience, even if you couldn’t fix your mistakes without pain and suffering due to possible memory shifts. When we were offered the opportunity to use Turbo Pascal as an environment for launching assembler instructions, we were immensely happy; the process of studying and developing applications in Assembler was greatly simplified. I feel the same when, instead of tormenting myself with the Tempest configuration, I can test my cloud with a couple of simple commands.
In most cases, using Tempest is difficult and time consuming. In addition, Tempest does not support a number of functionalities for load testing, such as load simulation and presentation of results in graphical form. Rally
can solve these problems - the OpenStack project, which is capable of many things, in particular: creating environments for verification, testing performance, as well as OpenStack benchmarking (including using Tempest).
If we talk about Rally + Tempest, then Rally automates the process of installing, configuring and running Tempest tests. Unlike Tempest, which must be installed and configured separately on each cluster, once installed, the Rally service can check a huge number of clouds - you just need to add information about your clusters in Rally, and you can easily switch between them later. The Rally benchmarking engine is able to automatically run tests, simulating user load. The results of both verification and benchmarks are stored in the Rally database. You can view them immediately, or after a certain period, to compare how changes in the configuration of your cloud affect its operation.
Next, we will look at how to use Rally and Tempest, but before moving on to the most interesting, we list the main reasons for integrating Rally with Tempest:
1. Most OpenStack projects use CI / CD systems based on Tempest to automatically test new changes. Based on this, if there is an opportunity to make life easier for developers, namely to allow them to use Tempest more actively and conveniently, then we must do it!
2. Load testing using Tempest tests:
1. Many tests have already been developed within the framework of Tempest, so there is no need to duplicate them in Rally. The less code duplication, the better.
2. Verification should not be limited to no-load testing. Rally can simulate the load for Tempest. In other words, the benchmarking mechanism in Rally can run Tempest tests with a different number of (simulated) active users.
3. Debbuging Tempest and OpenStack. When a race condition is detected in gates, developers will be able to run the same test (or test suite) under load in their environment, which will make it easier to identify the source of this kind of error.
You can talk a lot about the cool and promising features of Rally, but you will never know how good Rally is until you test it in action. So, probably you are ready to move from the "lyrics" to practice. Then let's get started!
Rally Installation
Not surprisingly, to use Rally, you must first download it ( github repository ), and then run the install_rally.sh installation script.
Adding your cloud to Rally
After a successful installation, you should add a cloud to Rally, which will be tested. This operation can be performed in two ways. One of the options for initializing a cluster in Rally is to use the OpenRC of your cloud. For example, like this: The
second method is to describe the cloud in the json file , which should be passed to the rally deployment create using the –file argument: By executing the
rally deployment list command, you can see a list of all initialized deployments.
Checking your cloud
Using the `rally deployment check` you can check the correctness of the added deployment, as well as see what services are available in your cloud from Rally:
Preparing Tempest for Your Cloud
As mentioned earlier, Rally can simultaneously work with a huge number of clouds, therefore, for each cloud custom Tempest. To reduce the time it takes to clone a Tempest repository, Rally clones it once and then uses the local repository, customizing it for each cloud.
Tempest preparation (cloning the repository and installing a virtual environment with all the dependencies) is done with the rally-manage tempest install command.
Launch Tempest
The start of automatic cloud verification (in other words, cloud verification) is implemented using the rally verify start command; it requires only one argument: the name of the test suite. If this argument is not specified, smoke tests will be performed (by default). The result of this command is similar to the result issued by Tempest.
Valid names for test suites are: full, smoke, baremetal, compute, data_processing, identity, image, network, object_storage, orchestration, telemetry and volume.
List all Tempest results
The results of all verification are stored in the Rally database, so you can view and compare them at any time. Lists of running checks will be displayed on the screen using the `rally verify list` command.
View the results of a specific verification
You can view the results of a specific verification with two commands: rally verify show and rally verify detailed. The latter complements the output of the first command with tracebacks of tests that failed.
Benchmarking with tempest tests
In addition to installing, configuring, and running Tempest, Rally can model user workloads using Tempest tests. The results of stress testing are also saved and can subsequently be displayed.
Run the benchmark and simulate the load (from 2 to 10 users) for the following test scenario: tempest.api.identity.admin.v3.test_domains.DomainsTestXML.test_create_update_delete_domain.
After the test is completed, using the rally task plot2html command, we generate an HTML file with the results. Information and examples of how the input data are correctly set in the JSON / YAML files used to define this task can be found in the Rally repository (in the rally / doc / samples / tasks directory) or on the OpenStack wiki page .
With 2 active users:
With 6 active users:
With 10 active users:
Conclusion
When using clouds in production, you have to deal with constant changes. The question arises: how to check each change? Tempest is a good but cumbersome solution. Rally, among other things, can simplify its use.
Rally provides a mechanism for stress testing OpenStack, and now, thanks to integration with Tempest, the library of test scripts has become huge. Will your fix speed up any operations? Do you have the same fast cloud as your competitors? Why not figure it out with Rally ?!
Original article in English .