Second billing, marketplace and sandboxes for Big Data: what can test environments in the cloud

Any software development company needs test environments that are close to the production environment. This is especially true for boxed software, which has a long release cycle.
Many problems building test environments solves their placement in the cloud. We will tell you about the testing capabilities on our Mail.Ru Cloud Solutions (MCS) cloud platform . But part of what we tell is true for any cloud.

The complexity of setting up a test environment

Before we talk about the possibilities of test environments in the cloud, we will talk about the difficulties that companies face in the process of software testing.

Different Git branches - different environments

Most software development companies use version control systems. The most common of these is Git , which is used by 87% of developers (a RhodeCode survey on Twitter ).

The best practice in Git are the so-called feature-branches (branches), when for each new functionality a separate branch in the repository is allocated. This approach allows developers not to be “pushed by elbows” and makes changes more independent, but requires the deployment of many dedicated environments for testing individual features.

Underutilization of computing resources

25% of physical servers are “zombies” who consume electricity but do nothing useful. And many IT specialists cannot say what 15–30% of the servers installed in their company are doing.

So here. On-premise iron for test environments here is no different from any other and, as a rule, poorly disposed of. And there can be no other way. When you buy a fleet of server hardware, you reload on resources in case of increased consumption and number of test environments. As a result, test environments are idle at night and on weekends “just in case”, and you just pay for the electricity and cooling consumed.
Simple application of virtualization in your private cloud does not solve the problem of inflexible scalability and under-utilization of equipment.

Difficulty setting

The test and development environments are not just a few virtual machines. It typically includes an application server, a database server, plus message queue servers and cache servers.

Developers find it difficult to raise and configure such infrastructure on their own, if they do not have the appropriate expertise, it may take several days. For experienced admins this is easy, but not interesting at all. Plus, bureaucracy is superimposed, and the process of allocating resources to creating a test environment can be very long.

Another problem is that a full-time system administrator often does not have expertise in deploying specific test environments, and the company is forced to attract expensive consultants from outside. An example is a sandbox for big data based on Hadoop, Spark, HDFS or Airflow. Deployment of such an environment takes at least a week for a specialist qualified in the BigData area and not less than a month for an ordinary system administrator. A similar story with Kubernetes clusters: building a test cluster close to the production configuration takes at least a few days.

As a result, the company is forced to turn to a third party - to attract consultants. However, finding big data experts is hard, and administrators of such systems can be counted almost on the fingers. Yes, the DevOps engineer can cope with all the tasks for deploying test environments, but he also doesn’t have it in every company, and finding a good expert in this area is no easier than a BigData specialist. In 2016, the job search site indeed.com noted that companies are looking for a DevOps engineer longer than any other specialists.

How the cloud solves these problems

Second billing only for consumed resources and instant scaling

The most obvious advantage of cloud services is the ability to rent the cloud infrastructure of any configuration (one server or a whole cluster) for any period. On our Mail.Ru Cloud Solutions (MCS) platform, billing is per second - you only need to pay for the computing resources that were actually used.

Unlike bare metal, private cloud and most Russian cloud providers, MCS does not rate the resources of stopped virtual machines (RAM, CPU): you only need to pay for the occupied disk space. For example, a test configuration of 20 CPU, 40 GB of RAM and 1 TB HDD costs 24,800 ₽ / month, and payment for disk space - 7,000 ₽ / month. For example, if automation means to stop such an environment at 9:00 pm and deploy at 9:00 am, you will pay $ 15,900 - one and a half times less than with a round-the-clock lease.

In our experience, the total savings from testing in the cloud can reach 60–70% compared to testing in the on-premise infrastructure.

The ability to instantly scale resources in the cloud up and down allows you to quickly deploy environments only at the right time and dispose of rented capacity by almost 100%. The deployment time of the test environment is reduced by several days, or even months, compared to on-premise.

Resources are allocated by the developers themselves.

Using self-service portals, developers can set up test environments in the cloud and allocate resources for them without asking the administrators. Even this simple action can shorten the time-to-market product by several weeks.

API for automatic creation and destruction of test environments

The cloud allows you to create short-lived environments on the model " just in time ". C via REST API and CLI can be massively deploy test environment, stop, update and delete them. In this way, you can configure the full life cycle for test environments and keep hundreds of environments running. Thanks to instant cloud scaling and per-second billing, flexibility and reduced operating costs are achieved.


Another advantage of cloud test environments is the integration with CI / CD tools. These solutions (for example, Jenkins) contain plug-ins that allow you to dynamically create test environments in the MCS cloud during the build. The test environment can be activated right before running functional or regression tests. If the tests were successful, the environment will collapse automatically, if something is wrong - the environment will continue, so that developers can reconnect and understand the cause of the regression.

Ready building blocks for test environments

Quickly deploy your test environment to the Mail.Ru Cloud Solutions platform is
assisted by PaaS, for example, Kubernetes containers and Databases in the cloud. Kubernetes is developing a service directory with hundreds of applications.

From the catalog, you can deploy ActiveMQ, RabbitMQ, Kafka clusters, log monitoring and analysis systems, various CMS and databases in a few minutes. Unlike the popular Docker Hub , which contains only individual application templates, the Kubernetes Service CatalogThere are high-level templates that simplify integration. From templates, you can deploy pre-configured application components (message queue, service discovery, databases, application servers, CI / CD tools, caching servers, blockchain tools, and more), without having to configure the virtual machines on which they are deployed.


Applications available in the Kubeapps marketplace.

Sandbox for Big Data

On MCS, you can deploy test environments for big data applications. Using the PaaS Big Data service, you can create Hadoop, Spark, HBase and Airflow clusters. The deployment process is fully automated, and it saves weeks of time compared to self-tuning.

An additional advantage here is given by second billing and instant scaling. The ability to create scalable test environments for short periods reduces the company's costs of maintaining an analytical IT infrastructure. Savings can reach 80% compared to on-premise.

Integration with Infrastructure-as-a-Code

Unlike proprietary solutions based on VMWare, the MCS cloud platform is based on OpenStack open source software, which has full integration with various tools: Terraform, Ansible, Puppet, Chef.

Terraform is built in the spirit of Infrastructure-as-Code (IaC), when the process of setting up the infrastructure is designed like writing code and is more familiar to developers. Terraform is difficult to use in a private cloud because it does not have full integration with VMware. In the MCS cloud, companies can use ready-made examples of working with Terraform (they are in this GitHub repository ).


Create a test environment using Terraform.

Remember

• Testing in the cloud is becoming a natural practice in the IT industry and has already been applied by a wide variety of organizations: from state universities to large IT companies .
• Building test environments in the cloud helps to save due to the possibility of instant scaling and automation through APIs, which provide almost 100% utilization of rented infrastructure.
• Additional services on a cloud platform and templates on Kuberntetes containers are ready-made building blocks that do not need to be customized.
• Services that are difficult to set up, such as Big Data processing tools, are easier to rent in the cloud in the form of pre-configured templates. You will save weeks by avoiding diving into settings.
• MCS is built on OpenStack and fully integrated with Terraform and other DevOps tools.

All these tools can be tried for free in the Mail.Ru Cloud Solutions platform. Until the end of November , this link with the ILOVEHABR promo code can add 1000 rubles to the account and test-test-test.