“Correct” Team Structure for DevOps / Raiffeisenbank Blog

Hello, Habr! For the second year in our bank, we have been discussing in theory and practice how to properly organize our Dev and Ops teams. Recently, the discussion, reinforced by a new portion of practical experience, entered the next round, which prompted me to the next search for ideas and arguments.


As a result, I came across one very curious material that impressed me so much that my desire to share it with the widest possible audience was overcome by laziness and lack of time for translation. Even despite the complete despondency that every time catches up with me when trying to find an adequate Russian word for the term “Silo”. The silo? German Oskarovich, I remember, used the word "well" ... After consulting with his colleagues, I decided to dwell on the term "plot".

What did I like about this article? First of all - the variety of options offered. No “do just that, and you will be happy”, which can often be heard from evangelists and consultants. Here are 9 “right” and 7 “wrong” options that you can try on for your organization, combine in different ways and if not get something workable, then at least a couple of ideas “to try”. Secondly, these stories are clearly taken from the real world of the “bloody enterprise” - most of them offer DevOps models not for a five-person startup who is doing everything right from the very beginning, but for a large office with its own legacy, complex organizational structure and intricate human relationship. After all, as we know from the recent Habr hit, even the most advanced organizations become alike upon reaching a certain size.

So, Matthew Skelton and Manuel Pais, article What is the best team structure for developing DevOps? . The first version appeared back in 2013, since then the authors have supplemented and revised a lot.

Anti-Types of DevOps Topologies

It is always helpful to have an idea of ​​bad practices that we may call “Anti-types” (rather than the ubiquitous term “anti-pattern”)

Anti-Type A: Plots Dev and Ops

This is the classic separation scheme between Dev and Ops on the principle of "throw software through the wall." In practice, for Dev, this means an earlier recognition of the fulfillment of tasks (“done” means “made a feature” rather than “working in battle”), but the software’s working ability suffers because Dev does not know what is really happening in industrial environment, and Ops do not have the ability or time to engage Dev in solving problems until a new version of the application is released for users.

We have all heard that this topology is not the best, but I personally think that there are even worse options.

Anti-Type B: Plotting DevOps Commands

As a rule, this anti-type arises when a certain senior manager decides that “we need a little of this DevOps” and instructs to create a “DevOps team” (possibly even consisting of “DevOps people”). Such a team will very quickly shield itself with a separate plot, pushing Dev and Ops away from each other, so that each of them can protect their corner, skills and tools from these “crooked developers” or “stupid admins”.

The only situation in which this model makes practical sense is when a separate DevOps team is created for a limited period of time, for a maximum of (for example) 12-18 months, with the original goal of bringing Dev and Ops closer together and with a clear statement on eliminating the DevOps team by expiration of this time. In this case, the DevOps model, which I call Type 5 , is implemented .

Anti-Type C: Dev-Ops is not needed

This model arises from a combination of naivety and arrogance of developers and their managers, especially when launching new projects or systems. Assuming that IT Ops is an image from the past (“we now have a cloud around us, right?”), Developers often underestimate the complexity and importance of operational skills and activities, and begin to believe that they can exist without them, or execute them in "free time.

Anti-type C is likely to turn into Type 3 (Ops as IaaS) or Type 4(DevOps as an external service) when the released software begins to create a real operational load and absorb the time allotted for development. Only if this team realizes the importance of IT Operations as a discipline can it avoid pain and unnecessary operational errors.

Anti-Type D: DevOps as a Tooling Team

In order to “become DevOps” without sacrificing the current productivity of developers (that is, implementing functional features), a DevOps team is created inside Dev to provide Dev with a whole set of tools - the development pipeline, configuration management system, an approach to managing environments, and so on. At the same time, Ops continue to work in isolation, and Dev - "throw software over the wall."

Despite the fact that the DevOps team in such a scheme can bring real benefits through the introduction of tools, the final effect will be limited. The fundamental problem remains unresolved: the lack of early involvement of Ops and the interaction of functions throughout the entire software life cycle.

Anti-Type E: Rebranded Sysadmins

This anti-type is typical for organizations with low engineering maturity. They want to improve practices and reduce costs, but they are not able to see in IT a key success factor for business. Since the benefits of DevOps to the industry are (already) considered obvious, they also want to “do DevOps”. Alas, instead of solving real problems in the structure or relationships of the teams, they choose the wrong path by hiring the "DevOps engineers" in the Ops team.

The term “DevOps” is used to rebrand the role of system administrators; no real changes in the culture or organization of processes occur. This anti-type is becoming more popular as indiscriminate recruiters join in the search for people with automation skills and DevOps tools. In reality, it is people’s communication skills that allow DevOps to work.

Anti Type F: Ops Inside Dev

This organization does not want to have a separate Ops team, so the Dev team is responsible for infrastructure, environmental management, monitoring and so on. With this approach, in projects or product teams, operational tasks become victims of resource constraints or low priorities, which, in turn, leads to poor-quality, raw products.

This anti-type demonstrates an underestimation of the importance of the role and skills of effective IT Operations.

Anti-Type G: Plots Dev and DBA

This is one of the varieties of Anti-Type A, which is very common in medium and large companies with many old systems tied to centralized databases. Because these databases are business critical, a separate DBA team, usually within the Ops framework, is responsible for administering, configuring, and recovering from failures. Such a scheme is logical. But if at the same time the DBA team becomes a guardian of the border and slows down all possible changes in the database, this becomes an additional obstacle to regular software updates (acceleration of which is the main goal of DevOps).

Moreover, if the DBA team is not involved in software development at an early stage, all problems with data migration, database performance, and so on, are found only in the late stages of the life cycle, which, coupled with a heavy load on administrators, leads to constant fires in battle and increasing pressure on the DBA.

Types of DevOps Topologies

After talking about the bad options, let's look at those models that help DevOps work.

Type 1: Collaboration Dev and Ops

This is the very "promised land" of DevOps: the "smooth" interaction of Dev- and Ops-teams, where necessary - specialization is applied, where necessary - teams work together. In this option, there may be several separate Dev commands, each of which works on a partially independent product.

In my opinion, the implementation of Type 1 requires significant organizational changes and a high level of competence in the management of the IT organization. Dev and Ops should have a clearly articulated, clear and understandable common goal (for example, “Deliver reliable and frequent software changes”). People from Ops should feel comfortable working with developers on development-specific issues, such as development through testing (TDD) or version control. On the other hand, Dev should be seriously involved in operational problems, and also strive to get introductory from Ops, developing new solutions. All this requires significant cultural changes in comparison with traditional approaches of the past.


Applicability Type 1: organizations with a strong technological component
Potential efficiency: high

Type 2: Fully Combined Teams

When people from Ops are fully integrated into product development teams, we get a Type 2 topology. In this case, the difference between Dev and Ops is so minimal that all employees are fully focused on a common goal. In principle, this is one of the varieties of Type 1 , but with its own characteristics.

Companies like Netflix or Facebook that provide customers with a single, purely digital product use a Type 2 topology, but I think this approach has limited applicability when an organization provides customers with more than one product. Budgetary constraints and the need for context switching, typically found in organizations that produce multiple products, may force a greater distance between Dev and Ops (use Type 1 topology)

Type 2 can also be called “NoOps” because there is no separate or visible Ops command in this model (although the Netflix NoOps model is also similar to Type 3 (Ops like IaaS)).


Type 2 Applicability: Organizations providing a single, fully digital product
Potential Efficiency: High

Type 3: Ops as IaaS (infrastructure as a service)

For organizations with a more traditional IT Ops business unit that cannot or will not change rapidly, as well as organizations that use public clouds such as Amazon EC2, Azure and the like for all their applications, it can be useful to treat Ops as a team, which provides a flexible infrastructure on which applications are installed and run. In this model, the Ops command is similar to Amazon EC2, that is, the IaaS (infrastructure as a service) approach.

With this scheme, a separate team (possibly virtual) works inside Dev, which acts as the center of expertise for operational features, metrics, monitoring, server deployment, and so on. She can also take care of all communications with the IaaS team. This team is still Dev by nature, and it uses a standard set of practices such as TDD, CI, iterative development, and more.

To simplify implementation, the IaaS topology has limited effectiveness (due to the loss of cooperation with the Ops team), but makes it possible to get a result faster than when trying to directly implement Type 1 (which can be the next development step).


Type 3 applicability: organizations with several different products and services, with a traditional IT Ops division; Organizations whose applications run fully in the public cloud
Potential performance: Medium

Type 4: DevOps as an external service

Some organizations, especially small ones, may not have financial resources, experience or qualified personnel in order to successfully cope with operational tasks in relation to their software. In this case, the Dev team can find an external service provider such as Rackspace to help build a test environment system, automate the infrastructure and build monitoring, as well as give advice on non-functional requirements that need to be implemented during development.

This option can be useful for small organizations or for teams that want to learn in practice about modern approaches to automation, monitoring or configuration management. In the future, as the organization grows and people appear in it with a pronounced focus on operational activities, the organization will be able to move to a Type 3 or even Type 1 model .


Type 4 applicability: Small teams and organizations with limited experience in IT Operations
Potential Efficiency: Medium

Type 5: Lifetime DevOps Team

This model looks the same as the Anti-type B (“Plot” DevOps), but the tasks and life span of the DevOps team are fundamentally different in it. A temporary DevOps team is created with the mission of bringing Dev and Ops closer together, ideally bringing them to Type 1 or Type 2 , and then self-eliminating due to uselessness.

The members of the temporary team initially work as “translators” between Dev and Ops, offering, on the one hand, crazy ideas such as stand-ups and Kanban boards for Ops teams, and on the other hand, together with a team of developers thinking of a low-level “kitchen” like settings balancers, SSL offloading, or managing network controllers. If enough people begin to see value in the convergence of Dev and Ops, then the DevOps team gets a real chance to achieve their goals. Important: a long-term responsibility for the implementation and application performance should not be "hung" on a temporary command, otherwise the scheme will quickly turn into anti-type of Bed and .


Applicability Type 5: This is a precursor to Type 1 , but there is a risk of turning into anti-type of Bed and .
Potential Efficiency: Low to High

Type 6: DevOps Evangelist Team

Within organizations with a large gap between Dev and Ops (or with a tendency to widen this gap) it may be useful to create a DevOps team that will support communication between Dev and Ops. This is a version based on Type 5 , in which the DevOps-team exists in a constant mode, but its task is to facilitate communication and interaction between Dev- and Ops-teams. Members of this DevOps team are often called evangelists because they spread knowledge about DevOps practices.


Type 6 Applicability: Organizations with a tendency to distance Dev and Ops. Danger - drift towards anti-type Bed and .
Potential Efficiency: Medium to High

Type 7: SRE Team (Google Model)

DevOps generally recommends involving Dev teams on duty on the phone, but this is not necessary. In fact, some organizations (including Google) use a different model, with an explicit ritual of software transfer from the development team to the team that is responsible for the operation - SRE (Site Reliability Engineering). In this model, Dev teams must present evidence to the SRE team (logs, metrics, and so on), clearly demonstrating that the transmitted application is reliable enough to support SRE.

What is important - the SRE team may not support an application that does not comply with SRE standards, asking developers to redo certain “features” before the application goes into commercial operation. The interaction between the Dev and SRE teams revolves around operational indicators, but as soon as the SRE team is satisfied with the quality of the application (it is it, not the Dev team), SRE begin to support this software.


Type 7 Applicability: Type 7 is applicable only to organizations with a high engineering culture and organizational maturity. Otherwise, there is a risk of getting Anti-Type A , in which SRE / Ops commands simply receive and follow the instructions for installing the software.
Potential efficiency: low to high

Type 8: Container Based Collaboration

Placing the application runtime in a container eliminates the real need for Dev and Ops collaboration. In this case, the container serves as a delimiter of the Dev and Ops liability zones. With a high level of engineering maturity, this model works well, but if Dev begin to ignore operational issues, then this option may turn into a classic “we are against them” confrontation.


Type 8 applicability: Containers may work fine, but this option can mutate into Anti-Type A if the Ops team expects it to support anything that the Dev team will throw in to it.
Potential efficiency: medium to high

Type 9: Dev-DBA Collaboration

This option is a response to the situation described in the anti-type of the G . To close the gap between Dev and DBA, some organizations bring the DBA team closer to a separate Dev team specializing in refining the central database. This helps in communication between people who are accustomed to looking at the database as a development object in which the application data is stored, and those who perceive the same database as a colossal amount of complex business information.


Type 9 applicability: Suitable for organizations where multiple applications work with one or more centralized databases
Potential Efficiency: Medium

P.S. We (Raiffeisenbank) ourselves agreed this summer to consider Type 1 (as a basic option for everyone) and Type 2 (as more "advanced" where it turns out to be targeted ). Life will show how this choice was right.

PPS And how are you? What types of DevOps do your organizations have?