Managing Dependencies in a Complex Agile Environment

Translation of the article "Dependency Management in a Large Agile Environment" .

Short review


Salesforce.com’s development department includes more than 30 Scrum teams working together on common code in the same version control branch. This article describes the methods used by salesforce.com to scale the Scrum approach and to manage team interactions.

1. Introduction


In October 2006, the salesforce.com development and development (R&D) department began a massive transition from a waterfall model to Scrum-based agile methodologies. At that time, 10 months passed from the previous major release, and the release date of the new one was postponed five times already. Many were upset that the product was rarely produced and with serious delays. We did not wait for the release to be completed, reorganized the existing teams into Scrum teams and, using Scrum processes, released the release in February 2007. Since then, using our new flexible approach, we have already released five major releases (lasting 3-4 months) of our set of SaaS applications and the Force.com platform. Each of them took place exactly on the scheduled day.

During the reorganization, we followed the Scrum recommendations for individual teams, but did not pay much attention to the interaction between the teams. Forming teams, we tried to minimize dependencies between them, however, the code did not change overnight, so many relationships remained. Pretty soon, we introduced Scrum-of-Scrum meetings. These meetings helped discuss issues and state of affairs, but meetings alone were not enough. Working on the last five releases, we tried and polished additional approaches that improve team interaction. Later in the article we will talk about some of the difficulties with dependency management and how we overcome these problems.

2 Team structure


Product development at salesforce.com consists of three divisions: Applications, Platform and Core Infrastructure. These three units contain a total of more than 30 Scrum commands. Each Scrum team has developers, testers and a Product Owner, as well as a Scrum Master (usually it’s the head of developers or testers or the Program Manager). For complex features, Scrum teams also attract representatives of other functional teams, for example, System Testing, Documentation, UI Design, Usability, Technology Operations, Release Engineering.
Typically, all Scrum teams work on the same release in the same code branch. Our products are interconnected, so the features of different teams can be very closely intertwined - both functionally and technically. From the point of view of technical implementation, many teams use a common code. In terms of functionality, teams working on features for the same end user should work closely to create a harmonious and consistent user experience.

3 Why is dependency management difficult?


When so many teams work on one release, introducing common features and changing the general code, it is impossible to anticipate all the problems, surprises, changes, failures and successes that will occur during the development process. This unpredictability is one of the reasons why dependency management is so difficult. Later in this section, we note a few more reasons. Any decisions should be directed towards these fundamental conflicts and challenges.

3.1 System Complexity

Identifying dependencies is the first step in managing them. Like many mature products, salesforce.com systems are too complex for one person or team to be able to know and remember all the relationships. To identify and recognize addictions, we increasingly have to rely on the collective knowledge of many people. It is imperative to combine such knowledge and connect people with information with people who need it. A key role in determining the dependencies and consequences of proposed changes is played by experts with a broad understanding of the system. However, as the system grows and becomes more complex, it is difficult to avoid specialization and fragmentation of knowledge. In addition, with the current number of changes, it is impractical to ask experts to check the details of each new feature. Need a way to spot changes with a high probability affecting (or depending on) the work of other teams in order to pay special attention to such changes. Of course, this is not so easy, because sometimes relatively small changes seriously affect the entire system.

3.2 Conflict of priorities

Dependencies can lead to conflicts between teams. If one team wants another team to do something, then Product Owners of these teams discuss and prioritize this work. If, based on the results of the discussion, it is possible to agree on the deadlines, then in the future to manage this dependence is quite simple. If a clear agreement is not reached, or if the priority of work is still low, then the uncertainty and level of risk around this relationship increase.

Conflicts also arise when one team makes a change that adds work to another team. For example, when one team changes the architecture, so the rest need to adapt to these changes. Change does not always bring obvious benefits to other teams, so sometimes people are offended and resent the appearance of additional work, especially when it appears suddenly. We try to identify such dependencies in the early stages, but some of them inevitably appear later - due to changes already made or after a review of priorities.

3.3 Changes in plans

One of the benefits of Scrum is the ability to change team priorities for each sprint. However, this complicates the work with dependencies. You can’t just identify and think through all the dependencies at the very beginning of the release cycle. Relationship analysis is more like a continuous, ongoing process. As work progresses, dependencies appear and disappear, while teams refine the details of what needs to be done. Therefore, the process of managing such dependencies must be dynamic and active.

3.4 Short and overlapping release cycles

In short release cycles, there is less time for coordination between teams, so the relationships and their influence must be identified at an early stage. At salesforce.com, release cycles intersect, in the sense that planning for a new release begins before the previous one is completed. This was done intentionally, as some teams have already freed themselves and may begin work on a new release. However, the rest of the teams may be late in planning and skip the discussion of dependencies for the next release. This is a serious problem, given the importance of collective knowledge to clarify relationships.

4 Special approaches


In this section, we will discuss the specific approaches that we use to manage dependencies and deal with the above issues. Common strategies used in these approaches:

  • Make it well known - release plans, relationships, projects, build status, etc .;
  • To organize platforms for discussions and knowledge sharing between teams;
  • Encourage self-organization and decentralized relationship management;
  • Automation, automation and automation again.


4.1 Launch Release

About a month before the release of the current major release, we arrange a kick-off meeting for the next release. The meeting is attended by all employees of the company. Vice-presidents of business units tell in general terms what features each team plans to work on in the new release. This meeting helps you manage your relationships. First of all, it encourages teams to prepare their initial release plans for the intended date. If a team has incomplete work from the current release, then this team begins to delay planning for the next release; in this case, the meeting helps to plan ahead of schedule. It is difficult to identify dependencies and discuss obligations with a team that has no idea what it will do.

In addition, after the meeting, the team’s plans for release become well known. We strive to achieve high awareness of what all teams do, because we believe that it makes people think and discuss relationships. All R&D leaders participate in the kick-off meeting, helping to attract people and focus on release planning. A high level of participation helps align expectations and inform the board about release plans.

Of course, plans can and do change in the process. Not so long ago, we began to review the updated launch meeting presentation during our monthly sprint reviews. An updated presentation shows which features were excluded or added, and it helps to maintain clarity and publicity of plans during the release.

4.2 Relationship Session

Having prepared initial release plans, teams can more closely discuss relationships, and many such discussions arise spontaneously. In addition to this, we considered it very valuable to conduct a more formal session to identify relationships, when representatives of all teams work together. It is most useful to conduct such a session shortly before the launch meeting of the release in order to improve the quality of the plans announced at this meeting.

A relationship identification session is fairly straightforward. Representatives (most often Scrum Master or Product Owner) of all Scrum-teams gather in a room with a large board. In the first part of the session, all participants simultaneously draw two dependency diagrams between the teams on the board. One diagram depicts teams that need another team to do something for them. The second diagram depicts teams doing something that will affect the work of other teams. We use a few simple rules for these diagrams:

  • A circle indicates a team;
  • The arrow between the circles indicates the relationship. In the first diagram, the arrow is drawn from the team performing the work to the team that needs the result of this work. The second arrow indicates a team that is affected by the work of others;
  • The label on the arrow describes the essence of the dependency;
  • If the dependency is agreed (that is, the other team agrees to do its part), we draw a frame around the label.

At first confusion reigns, while everyone draws his own relationships, and observers ask questions and point out additional dependencies. After about twenty minutes, the chart stabilizes, and the comments run out. After that, we ask the representative of each team to briefly talk about their dependencies.

In a very short time, we create a complete picture of the relationships in the release. In fig. Figure 1 shows a typical session result after being rendered more readable. Hot spots are immediately identified (that is, teams with a large number of dependencies). Also visible are teams that have inconsistent dependencies. Such teams are most at risk of not achieving their plans until their relationship is agreed.


Fig. 1. Cross-Dependency Chart Example

4.3 Open discussions

Recently, within a week after the kick-off meeting, we have been holding open meetings, the main purpose of which is to create a platform for discussing issues and problems related to the team’s plans for release. We ask that at these meetings there should be at least one representative from each Scrum team, otherwise they are not required to attend. Usually, at the beginning of an open meeting, participants suggest topics for discussion. Then, within 45 minutes, discussions are held on topics in groups, and then groups tell the rest about the results of the discussion. Most often, participants want to learn the details of new and useful functionality, as well as about changes that will affect other teams. Participants in open meetings noted the instructiveness of such discussions, in addition, people found it interesting to communicate with colleagues from other teams, with which they do not interact in normal work. Open meetings are still a new format for us, so we try different options to understand what works better, for example:

  • To suggest ideas before the meeting;
  • Involve high authorities;
  • Change the number and duration of discussions.


4.4 Design Overview

The main goal of functional design review meetings is to increase the overall level of quality and usability of our products by:

  • the collective wisdom and creativity of our product teams;
  • improved design consistency between our products;
  • synergies arising from the exchange of knowledge between teams.

These cross-team meetings focus on the design of functionality and user interaction for features that we consider to be key, most complex, or especially at risk. At such meetings, we strive to analyze and identify previously undetected inconsistencies and dependencies.

There are two main types of such meetings. In the early stages of the release cycle, participants focus on design concepts and strive to achieve coherence and effective interaction between existing and new features. At this stage, Product Owners and User Interface Designers (UIs), and occasionally representatives of other functional teams such as development and QA, usually participate in meetings.

Around the middle of the release cycle, the composition of participants is changing: now they are mainly representatives of QA and partly development, and discussions are focused on implementation details. These meetings provide understanding and clarity regarding:

  • Additional relationships between teams;
  • Additional integration testing, which must be included in the plans;
  • Risks of individual features, and sometimes - the need to restrict access to features;
  • The specific tasks of deployment.


4.5 Virtual architecture team

The Virtual Architecture Team (VAT) is “virtual” because it includes developers from all Scrum teams. Team members work in VAT without interruption from their main responsibilities as part of a Scrum team from the Applications, Platform or Core Infrastructure business units.

VAT is responsible for supporting and developing the architecture of our software. The team defines a road map for the development of architecture, reviews major changes from an architectural point of view, determines coding standards to ensure code compatibility and support.

VAT manages the backlog of important architectural projects and the necessary refactoring. As the product develops, we have to redesign the features and get rid of the old non-optimal code. Sometimes developers do not want to change already running programs, even when they understand that the program code leaves much to be desired. Product Owners prefer to see how new features are developed, rather than how something already working is being reworked. To counter such moods, each of our business units is required to plan at least 20% of the time in each release for changes recommended by VAT.

VAT is collected twice a week for two hours to review the technical implementation of products and features created by Scrum teams. Teams working on the most complex release features are required to submit their VAT. The VAT gives the Scrum team feedback on how their technical solutions will affect the rest of the teams, and what developments of other teams can affect the feature. VAT focuses primarily on technical implementation, especially on the aspects of scalability and performance. If the feature design requires major changes, then the Scrum-team is obliged to re-submit the feature in the same release cycle and show the VAT how they changed their design.

4.6 Continuous Integration

Our web-based infrastructure for automated assembly, testing and sorting (triage) provides continuous integration of the build system and allows you to monitor the status of each line of code. This infrastructure is a key aspect that allows all developers and QA engineers to work with a common code base.

The main test suite built on the JUnit extension allows you to create functional tests using the Force.com API. In addition, we have a user interface testing framework that uses Selenium to automate test cases that need to be done through the UI.

Here are a few fundamental principles that define our approach to automation:

  1. Give developers quick feedback so they can see the results of their changes. Each commit starts building a build and executing a set of tests. In the event of an error, responsible engineers will immediately receive a failure message. The basic set of tests is completed in half an hour, and the developer receives the results of verification of the changes made by him. Periodically throughout the day, an extended set of tests is performed;
  2. Quickly repair builds and test failures . To do this, we have the role of “assembly wizard”, changing owners every week. Typically, this role is played by two closely interacting people - a development manager and a senior developer. The build wizard is responsible for checking the build results, tracking the test history and failures on various lines of code, as well as assigning bugs to specific developers for fixing. We have a special panel that displays various metrics based on the test results collected in the database (percentage of successful tests, build time, number of failures, etc.). The assembly wizard uses this data to prepare an assembly status report;
  3. Maintain a high level of coverage with automated tests . Typically, Scrum teams tend to cover 70-80% of the code with automated tests. One of our most important assets is a large set of autotests, which plays a key role in delivering quality releases on time every 3-4 months.


4.7 Scrum-of-Scrums

Every business unit hosts a Scrum-of-Scrums weekly meeting, and there is a Scrum-of-Scrum-of-Scrums. Sometimes we organize additional Scrum-of-Scrums for a group of teams that work closely together on a common goal. We tried out several different Scrum-of-Scrums formats. We started with the standard “4 questions” format, when teams reported what they had done, what they were going to do, what was blocking them, and whether they were doing something that affected the work of other teams. At first it worked, but soon became tedious and boring due to the large number of teams. Then we moved on to a more open format of self-organization, when participants proposed topics for discussion, writing them out on the board at the beginning of the meeting. This forced the participants to take responsibility for the content of the rally and led to more productive discussions. The issue of cross-command dependencies is often raised during Scrum-of-Scrums, especially in the early stages of development. A relationship recognition session is held during the Scrum-of-Scrums, and over the next two to three weeks this meeting often discusses changes in inter-command dependencies.

4.8 Status Reports

When a team switches to Scrum, written reports often become unnecessary, because clarity of state is now ensured by other things (sprint reviews, burndown chart, daily stand-up meetings). When we switched to Scrum, we decided to keep the lightweight weekly status reports prepared by each Scrum master. It looked redundant while we used the “4 questions” format for Scrum-of-Scrums. However, with the current open Scrum-of-Scrums format, weekly reports have become an important addition to this meeting. We do not discuss the status of each team during Scrum-of-Scrums, unless it is submitted as a separate issue, but this status is always available in the weekly report. The report identifies all dependencies, blockers and risks. Of course, such reports require a little extra effort from Scrum masters,

5 Conclusion


Salesforce.com has proven that Scrum scalability is feasible, and we are confident that we can continue to scale it as the company grows. As the number of teams grows, relationship management and team coordination become a daunting task. A robust build and automated testing infrastructure is critical. Awareness and synchronization helps a lot with the help of the kick-off meeting, relationship discovery session, Scrum-of-Scrums and lightweight status reports. Ultimately, this leads to effective communication, interaction and exchange of knowledge between teams. Approaches such as a virtual architecture team, open discussions, and Scrum-of-Scrums are designed to help communication and support collaboration.

Also popular now: