Testing at Openshift: Introduction

Hello dear members of the IT community. My name is Oleg, I work for a company that develops software. I am involved in manual and automated testing of Linux and Unix products and I would like to share the positive experience of automated testing in Openshift Origin .

The goals that I pursue :

1. To convey to the Russian-speaking community the features of working with Openshift Origin in the context of testing.
2. Talk about the advantages and disadvantages of testing in containers.
3. Aggregate and update your knowledge of Kubernetes / Openshift.

All material is presented in three articles :

1. Testing at Openshift: Introduction
2. Testing in Openshift: Cluster Internal Device
3. Testing at Openshift: Automated Testing

Note: I would like to immediately notice that the material presented concerns Openshift v3, and not Openshift v2 (when Red Hat had not yet started using Kubernetes as a kernel for its products and services).

Why Openshift Origin was chosen :

It so happened that after employment at the current place of work, it turned out that there was no automated testing in principle. After working for several months in manual mode, an understanding came about the need for two things: a tool for deploying and supporting test environments, a framework for writing tests.

At the initial stage, for the deployment and support of environments, an automatically updated Vagrant repository ( VirtualBox ) was organized , which updates and packs various OSs in an automatic mode. This became a help not only for testing, but also for development, because: virtual machines were configured according to the developed requirements; virtual machines were updated, the necessary tools were preinstalled; There were Vagrant scripts for deploying complex environments. It is worth noting that all Vagrant environments are deployed on the local machines of the development participants, and not in a dedicated IaaS , which was expected to cause workstation performance problems.

Time was won, it became more convenient to work, a certain standard and predictability appeared, but the main problem remained - a long deployment of test environments (full virtualization, various Linux distributions, additional services (which are involved in testing)). Deploying test environments could take tens of minutes, while the test itself took place in seconds. At the same time, the number of automated tests grew and the results had to wait longer.

The logical question was the organization of a separate IaaS in which all testing tasks would take place, but this approach was not ideal: full virtualization would still be used, IaaS would require financial investments to purchase hardware (a small fleet of weak workstations was present) .

The next step was the development of a fast (in the context of environment performance) CI / CD taking into account the new requirements, namely (basic, according to priorities):

1. Open source solution without any cost for the purchase of software.
2. Fast deployment of test environments. Connectivity and awareness of the existence of services between environments. Built-in support and control of test runs inside the environment. High density to meet paragraph 4.
3. An active community, a popular project with a predictable future, the reputation of the owner, the availability of documentation.
4. The lack of additional investment in hardware.

I will not bother readers with a detailed retelling of the thorny path to finding suitable products, but I want to familiarize myself with the list of reviewed software (February 2016):

1. Replacing full virtualization: Docker , LXC , OpenVZ , kvmtool .
2. Orchestration and cluster management: Kubernetes , Nomad , Openshift , Openstack , Swarm .
3. Jenkins Support : Docker plugin , Kubernetes plugin , Openshift plugin

Some tools were poorly managed or required too much support and maintenance, others were poorly integrated with Jenkins and did not provide the necessary functionality, while others were just starting their way. An unequivocal leader loomed Kubernetes and its derivatives (of which there are many), but in its pure form, the platform from Google was difficult to learn and deploy (developers are making great efforts to simplify the use of this platform).

In your own words about Openshift Origin:

Openshift Origin is primarily an Open Source software development and publishing platform based on the Kubernetes container orchestration platform. The platform extends Kubernetes with the help of special objects and mechanisms.

Cluster:

The cluster can be deployed or updated using Ansible scripts . Inside the cluster, both the built-in and external Docker register can be used. Cluster nodes can be assigned based on special labels to individual projects. There is a garbage collector and a custom scheduler . The cluster can be deployed inside Openstack and integrated with it to automatically scale and provide data storage. Environments can be aware of published services and other nodes using environment variables and DNS names. It is possible to check the availability of services through an HTTP / TCP request and / or through the execution of a command inside the container. Cluster Resources Can Be Quotedat the project level (processor, memory, number of objects, etc.). There is the ability to monitor the cluster at the level of containers and work nodes.

Data:

Temporary or permanent volumes, which are mounted directly in containers, can act as a method of data storage. Backend for these volumes can be: NFS, GlusterFS, OpenStack Cinder, Ceph RBD, AWS Elastic Block Store (EBS), GCE Persistent Disk, etc. There is a possibility (hostPath) of mounting the local directory of the working node on which the container is running.

Network:

By default, all communications are performed using Open vSwitch , but there is support for other network solutions through the Kubernetes plugin system. By default, all environments can communicate with each other, but isolation at the project level is possible. Supported DNS resolution of service names using the integrated SkyDNS service . Services published within the cluster may be accessible externally. Iptables takes care of all the isolation and NAT functionality .

Security:

Differentiation of user rights based on roles. Container isolation with SELinux (and more). Support for secrets that are used to access various resources. All communication between the working nodes of the cluster and the master is done through encrypted connections (a CA is created, client certificates are issued).

Control:

APIs are available for both Openshift itself and Kubernetes. A cross-platform console client is available . A web interface is available with which users can: work in isolated projects taking into account their authority, interact with containers running in the cluster, view created objects, track events, etc.

Conclusion:

Despite the abundance of technologies and products that are available on the market, finding a suitable tool is quite difficult. Balancing between the unification and integration of CI / CD processes, given the complexity of maintenance, tracking the chains of technologies involved, Kubernetes was found, and then Openshift Origin. Unlike the basic platform of orchestration with Kubernetes containers, Openshift brings the necessary elements for convenient and efficient work.