Mission feasible: how to develop DevOps in a company with many projects
A lot of articles have already been written about the benefits of implementing DevOps on Habré and other IT resources, and it is not in doubt. This is understandable: a creative mess with sometimes unobvious areas of responsibility of the “usual” development, where different people are responsible for the code, brunches, test benches, assembly and deployment and do not really want to climb into the clearing of colleagues, is opposed to a high level of organization .
In companies with embedded DevOps, developers have more support and can work more efficiently. But it’s not enough just to want to work “correctly”; you still need to somehow come to this. But here everything is not so simple.
More precisely, it can, but easily, this process can only go through in small companies where one or two projects are developed, or when all teams initially have a high level of development culture. But what about internal startups, where the idea often overtakes the physical and technical capabilities of the company?
When we at Positive Technologies decided to develop DevOps, we ran into dozens of teams that worked on projects, both public and non-public. The teams were very different in size, using different release models and a technological stack. All this did not allow to create a solution on the principle of one size fits all.
Moreover, resources are always limited and they were not enough to introduce DevOps approaches within each team and project. Some new approach was needed. And this is what we came up with.
We needed a set of elementary DevOps practices that we could quickly adapt to the tasks of any project. To do this, it was necessary to develop templates for assembly, deployment and test configurations, to ensure scaling of projects for many git-branches of individual components, as well as a large number of components and their versions.
We managed to solve these problems by transferring the Continuous Integration infrastructure to using a bundle of several tools:
In the course of numerous experiments on real projects, the TeamCity project hierarchy and a standard interface for all projects in this system were developed.
The processes were elaborated in detail:
By clicking on the picture will open in full size
All this is good, but there were questions about the implementation of DevOps in all projects of the company.
But what if we combine several distinguished specialists with various specialized knowledge into one group that can solve DevOps problems of the whole company?
So the idea of creating a "fire brigade" came up. Now, instead of inventing automation from scratch for each project independently, employees working on it can order the DevOps service.
As a result, we were able to significantly reduce the cost of training and starting work on the new methodology, as well as reduce the number of possible errors.
The topic of implementing DevOs is extremely interesting, various approaches are possible here. And it is extremely important to meet and share experiences. Usually such events are called “mitaps”.
We are going to essentially do something similar, but not quite in the usual format: we want to collect the “devs” and developers in a relaxed atmosphere in order to tell the story of building our own DevOps in fast track mode and listen to you, those for whom we did it : developers, automation engineers and testers.
Waiting for you at Op! DevOps! Friday, October 7th from 3 p.m. to 7 p.m. at Blacksmith Irish Pub
The event is free, but the number of seats is limited, so you need to register to participate. To do this, you need to register .
In companies with embedded DevOps, developers have more support and can work more efficiently. But it’s not enough just to want to work “correctly”; you still need to somehow come to this. But here everything is not so simple.
You can’t just take and develop DevOps (c)
More precisely, it can, but easily, this process can only go through in small companies where one or two projects are developed, or when all teams initially have a high level of development culture. But what about internal startups, where the idea often overtakes the physical and technical capabilities of the company?
When we at Positive Technologies decided to develop DevOps, we ran into dozens of teams that worked on projects, both public and non-public. The teams were very different in size, using different release models and a technological stack. All this did not allow to create a solution on the principle of one size fits all.
Moreover, resources are always limited and they were not enough to introduce DevOps approaches within each team and project. Some new approach was needed. And this is what we came up with.
First steps
We needed a set of elementary DevOps practices that we could quickly adapt to the tasks of any project. To do this, it was necessary to develop templates for assembly, deployment and test configurations, to ensure scaling of projects for many git-branches of individual components, as well as a large number of components and their versions.
We managed to solve these problems by transferring the Continuous Integration infrastructure to using a bundle of several tools:
- TeamCity - organization system Continuous Integration;
- GitLab - a storage system for the source code of component products;
- Artifactory is a storage system for assembled binary versions of components and products, caching proxies of external repositories for various package managers (nuget, npm, pip, rails).
In the course of numerous experiments on real projects, the TeamCity project hierarchy and a standard interface for all projects in this system were developed.
The processes were elaborated in detail:
- assembly and deployment to test stands;
- launch functional and other tests;
- assignment of quality labels to packages (test, stable, ready for distribution to customers);
- Publishing release builds to update servers
- delivery of assemblies and updates to production and many others.
By clicking on the picture will open in full size
What is the result
All this is good, but there were questions about the implementation of DevOps in all projects of the company.
But what if we combine several distinguished specialists with various specialized knowledge into one group that can solve DevOps problems of the whole company?
So the idea of creating a "fire brigade" came up. Now, instead of inventing automation from scratch for each project independently, employees working on it can order the DevOps service.
As a result, we were able to significantly reduce the cost of training and starting work on the new methodology, as well as reduce the number of possible errors.
The topic of implementing DevOs is extremely interesting, various approaches are possible here. And it is extremely important to meet and share experiences. Usually such events are called “mitaps”.
We are going to essentially do something similar, but not quite in the usual format: we want to collect the “devs” and developers in a relaxed atmosphere in order to tell the story of building our own DevOps in fast track mode and listen to you, those for whom we did it : developers, automation engineers and testers.
Waiting for you at Op! DevOps! Friday, October 7th from 3 p.m. to 7 p.m. at Blacksmith Irish Pub
More about the event program
No. | Topic | Presenter | Brief Abstracts |
---|---|---|---|
The first block of reports | |||
1 | Continuous Integration System Model at Positive Technologies [ slides , video ] | Timur Gilmullin | 1) The initial standard schemes offered by DevOps for all projects of the company: Build - Deploy - Testing - Promote. 2) Implementation of the scheme on the examples of our projects in TeamCtiy. 3) What have we come to. General Continuous Integration: Build - Deploy - Testing - Promote - Publishing - Delivery - Install & Update |
2 | SupplyLab - publication, delivery, deployment, licensing [ slides , videos ] | Alexander Pazdnikov | 1) Organization of an open system for managing the full cycle of delivery, deployment and licensing to the Customer. 2) Designing a publishing, delivery, deployment and licensing system - SupplyLab. |
3 | General concept of the server environment deployment system based on SaltStack [ slides , video ] | Dmitry Miroshnichenko | 1) Designing an update system. 2) About SaltStack. |
4 | Toolkit for creating product distributions [ slides , videos ] | Vladimir Selin | 1) What is a big product distribution? 2) Problem: a small number of people have knowledge about the product installation process. 3) Templates + DSL - the solution to all problems! |
The second block of reports | |||
5 | Organization of workflow in the TFS tracker [ slides , video ] | Alexey Soloviev | 1) TFS as a tracker: a brief overview of the features. 2) Structure of a typical Workflow: basic elements. 3) The difficulty of customizing WorkFlow in TFS. |
6 | vSphereTools - a tool for automating work with vSphere [ slides , video ] | Timur Gilmullin | 1) VIX API against pysphere. 2) vSphereTools is a set of scripts from DevOps to support working with vSphere and virtual machines. 3) Description of the tool, its advantages and disadvantages, possible improvements. |
7 | TeamCity and Character Server Integration [ slides , video ] | Alexey Soloviev | 1) What is a debug symbol server, its purpose. 2) Debugging information (debugging symbols) - information that the compiler generates on the basis of source codes. Contains information about the source file names, variables, procedures, functions. 3) Debug information server - a server whose main purpose is to store debug information, its indexing and providing access. |
8 | Tools for conducting competitive analysis of software products [ slides , videos ] | Vladimir Sofin | 1) What is the competitive analysis (CA) of software products? 2) Methodology and stages of spacecraft. 3) Difficulties in the implementation of the various stages of the spacecraft. 4) Tools for spacecraft automation. |
The third block of reports | |||
9 | Neuro-fuzzy classification of poorly formalized data [ slides , video ] | Timur Gilmullin | 1) Problems of automation of classification of weakly formalized (fuzzy) data. 2) Fuzzy sets and fuzzy measuring scales. 3) Neural network modeling for data classification. 4) The FuzzyClassificator tool and its implementation in the Company. 5) Automation of data classification based on TeamCity. |
10 | From simple to complex: we automate manual test plans [ slides , videos ] | Sergey Timchenko | 1) We look around - the usual process of auto-testing. 2) Remove unnecessary - realistic target process. 3) DataDrivenTesting - creating specials. tools for specific scenarios. 4) RobotFramework - what to do if there are too many simple scripts. |
eleven | Zabbix monitoring system in the development and testing processes [ slides , video ] | Alexey Burov | 1) A system for monitoring the resources of various departments. 2) Templates and roles of servers, differentiation of access and areas of responsibility. 3) ptzabbixtools - monitoring configuration on target servers. 4) An example of embedding a monitoring system in development / testing processes. |
12 | Automation of load testing in conjunction JMeter + TeamCity + Grafana [ slides , video ] | Ivan Ostanin and Sergey Tikhonov | 1) Description of the old process of littering test data: how it was before, what is good, what is bad. 2) Influxdb, as a repository of time-series data. 3) Zabbix - monitoring of load stands: Windows and Linux agents, active data collection, autodiscovery virtual machines in esx. 4) Grafana, as a way to turn graphics and dashboards into candy. 5) Automation of user load via web-UI using Jmeter, real-time statistics display, CI Teamcity. |
The fourth block of reports | |||
thirteen | CrossPM package manager: simplifying complex dependencies [ slides , video ] | Alexander Kovalev | 1) Difficulties in unraveling cross and nested dependencies. 2) CrossPM package manager. Its capabilities and examples of use. 3) Integration of CrossPM and Artifactory package storage systems. |
14 | Practical recommendations for using the TestRail system [ slides , video ] | Dmitry Ryltsov and Alexey Vasiliev | 1) Purpose of using TestRail. 2) The essence of the TestRail system. 3) Features of the project. 4) Our decision. 5) TestRail Integration & Customization. |
fifteen | TeamPass - management of access control for service passwords in a team [ slides , video ] | Dmitry Miroshnichenko | 1) What did not work in keepass? 2) Company specifics. 3) Solution. |
16 | DevOpsHQ community [ slides , video ] | Alexander Pazdnikov | 1) About the project DevOpsHQ - a community of automation engineers. 2) The pursued goals. 3) Suggested tools. 4) Suggested techniques. 5) Ready-made solutions. |
The event is free, but the number of seats is limited, so you need to register to participate. To do this, you need to register .