Experience transitioning from Waterfall to RUP methodology for implementing large IT projects



    How the need arose to move away from the classic Cascade Development Lifecycle Model


    In 2009, I was offered to choose and implement one of the “bastard” projects. Everyone received the prefix “Ghibliy” because they had already tried to take them, but nothing came of it.

    In large projects, the reason for failure often lies not in the professional level of the team, but in the Customer’s willingness to go to the end and get a return on their own efforts. The situation should be such that the goals of the project coincide with the short and medium term goals of the Customer.

    A great example: only having found itself under sanctions and an oil price of $ 50 per barrel (versus 110 previously) did the country's leadership switch from reasoning and unhurried body movements to active actions to develop a high-tech economy.

    So one of my Customers was ripe and I started to make a project for him to develop a new functional module of the Corporate Information System (ERP-system), which was to add 400 new users to the system and provide verification of 40,000 mortgage loans per year.

    After 2 months, we already had the first version of the Terms of Reference, which consisted of 200 pages of the main TOR and 200 pages of applications to it with descriptions of various forms and business algorithms. I must say that before that I saw the main TK for the entire corporate IP and at the time of implementation it consisted of only 600 pages. Therefore, even during the development of technical specifications for the module, the idea arose that with such a volume of requirements it was very likely that we would not take off, and if we take off, it would not be very fast.

    After the guys from our dedicated development center at EPAM announced the complexity of the implementation of 6500 hours and a duration of 1.5 years of development, it became clear that we definitely would not take off.

    The business process and procedures that needed to be automated were lively and constantly adapted to changes within the company and externally, both for market and for the requirements of the Central Bank of Russia.

    In a year and a half, those “pants” that we sew for them will be either small or not suitable for them at all, and this is money, development time and resources that have been wasted. It is enough to try to go through this once to understand that this is definitely not an option and career suicide within the company. After that, they don’t trust anything serious.

    This led to the conclusion that Waterfall is suitable for small projects and those where automated processes are not subject to constant changes. Something else was needed, involving early delivery and at the same time so that it was clear to everyone involved in the development process. Ideally Waterfall, but not Waterfall. It turned out to be the Rational Unified Process.

    Two words about context


    The company develops and develops its own corporate IP based on the Pivotal CRM platform. The reason for the independent development of CIS and additional functional modules for it is the lack of any alternatives on the domestic market and the expensive cost of acquiring Western counterparts in the 2000s.

    In addition to the company's employees, KIS employs employees of more than 200 Partner companies across the country (B2B). As mentioned above, development is outsourced by EPAM Systems. On the side of the company are business analytics and project management.

    What is wrong with the Waterfall methodology (features and disadvantages of the cascade model)


    This Mother methodology is for all subsequent and universal. According to this methodology, the first programs were developed.

    The fundamentals that hold this Methodology and are laid in it:

    1. From start to finish, the designed end result.

      (Architectural Result Design)


    2. Instructions for assembling the final result.

      (Project Implementation Plan)


    3. Consistent project implementation.

      Analysis, design, assembly, testing and commissioning


    4. Resistance to changes in the Result Project and Assembly Instructions during the project implementation.

      Project changes


    5. The project result is delivered as a single delivery at the end of the project.

      The dynamics of the assembly of the project result


    In order to implement a long-term project for Waterfall with terms of more than 1-2 years, you need to make assumptions about the business environment, the needs of customers and the company that will be at the time of completion of the project (assumptions related to the business goals of the Project). It is also necessary to assume and lay the estimated cost of resources for each year of the project's life (assumptions related to the project implementation plan).

    The greater the content of the project, the complexity and timing of implementation, the more assumptions are laid in the project, which may turn out to be both true and false. The most critical are the assumptions related to the business goals of the Project, if they turn out to be wrong then all the resources invested in the project, money and time are depreciated.

    Attempts to make significant changes to the Project Final Result during implementation require a cycle of redesigning the Final Result and the Assembly Instructions. Significant changes often do not allow the continuation of those already started according to the initial work plan and lead to a redo of what has already been done.

    For this reason, the project team and stakeholders resist significant changes in the project only if the circumstances and the risks that have played / revealed do not cast the whole project into doubt about the need to continue it. When changes are nevertheless made, they most often lead to additional costs and longer deadlines. The increased duration again gives rise to new requirements and new changes. It is like a vicious circle from which it is impossible to emerge victorious. From here comes the constant departure from the initial dates and budget.

    Therefore, this Methodology is suitable for projects for the construction of buildings and structures, such as a nuclear power plant, but is not suitable for the development of Products for rapidly changing environments in which they will be applied. For which the life of the Product in its original form without modification is less than 5 years.

    The idea of ​​iterative development is designed to eliminate the risks associated with assumptions, to provide an opportunity to quickly get feedback, reduce the scale of losses from incorrect business assumptions and make changes to the Project Final Outcome based on feedback received at the end of the next iteration.

    Benefits of Rational Unified Process


    The RUP methodology has been deliberately designed to develop large software systems.

    Novation No. 1 , laid in its foundation, is “Business Modeling” and the associated Use Case Scenarios.

    First, a business use case is developed, where the future system is a “black box” that satisfies all the needs of the User / Business. Based on it, system usage scenarios are developed that describe the functions of the system that will support the execution of the business scenario.

    The main task in developing corporate IP is to ensure the continuity of the supported business process. When the fabric of the business process breaks, then all the subsequent steps in the production chain arise, and sometimes the entire business worth tens of billions of rubles.

    Example of stopping trading on the Moscow Exchange
    An abnormal situation exists on the stock, currency and derivatives markets of the Moscow Exchange ... The last time the exchange suspended trading on September 1, but this only affected the Main Market section. This failure was the fourth in four months. - Lenta.ru

    Therefore, the development of corporate IP is based on the design of a new business process based on its use to replace one process with another and hundreds of people from a certain day began to work differently than they used to before. This is the main and main difference between the development of corporate IP from other types of software.

    Innovation No. 2 : Based on the selected system scenarios for the use of the system, architectural decisions are made and the components that will support business scenarios are identified and whether they are decided to be used together to support several business scenarios or remain tailored for a specific scenario. (Object Oriented Design and Programming)

    The presence of individual components allows them to be reused and replace one with another. The more components that are used together, the more likely it is to break an adjacent business scenario when making changes to the component. Also, the presence of common components limits the speed and scale of subsequent work on the development of the system. If one team is committed to changing some common component, the other cannot start work on changing it until the first finishes work and debugs its business use case.

    Novation No. 3 : The presence of dedicated scenarios for using the system allows us to divide them into primary and secondary. Secondary ones are based on the results of the execution of primary scenarios, which by and large are self-sufficient. This gives the opportunity to isolate the iterations and decompose the implementation of all the scenarios for them.

    There are no iterations in Waterfall, since it is sometimes incredibly difficult to separate functional requirements into self-contained blocks and implement them in iterations. I'm not saying that this cannot be done, it is possible, but most often the result of this will not be a working prototype of an IT solution, but some part of a future program. This doesn’t make it easier for the business, it will get a working prototype in a year, the presence of iterations here helps, but not much.

    This is what the RUP concept looks like in simplified form:


    Each business scenario is deployed in system use cases. Each system scenario is based on requirements for interfaces, algorithms, data and not functional (performance, availability time, response speed ...). Based on the identified requirements, the future architecture of the system and its functional modules (service subsystems) are determined. Subsystems are broken down into components. Components contain executable code.

    The main idea of ​​RUP is to issue in the first iterations a prototype of the future system with a portion of business scenarios ready for use / application. Since the delivery of a part of the Project Result (increment) is carried out at the end of each iteration, this allows you to begin to benefit from the use of intermediate versions of the Result and return investments in the project during its implementation.

    How we applied RUP


    Step No. 1 - I agreed with the manager on the use of a technical task template different from the one adopted by the company. The new TK is based on business and system usage scenarios. This decision was made earlier than the decision to work on RUP and for a number of reasons about which can be found in my article “What should be TK for corporate IP?” (Link at the end of this article in the section “What else to read”). 

    Step No. 2 - Designed the target business process and prepared the first version of the ToR. After we received estimates of the complexity and duration of the implementation, we decided that we would try to use RUP for this project. We broke down the target business process into 5 business procedures / 5 stages of the project. The first version of TK became a concept / roadmap for moving towards the target process. 

    Step No. 3 - Decided on automation strategies: move from the input or output point of the entire business process. 

    The first strategy "from the entry point" creates a voltage and forces you to automate all the other stages in the chain. In the subsequent stages of the business process, the primary data that they need to work ceases to come.

    The second strategy “from the exit point” does not have this tension, but it also does not have the data and history that are created and accumulated in the previous stages. We need regular transfer of the necessary information to CIS.

    Let's go on the first strategy. The problem of providing the subsequent steps with the necessary information was solved by uploading the data to Excel. We got an unexpected effect in the form of growing motivation for business users to completely switch from Excel to KIS, along with appropriate support and willingness to devote time to work with a business analyst to develop requirements. 

    Step No. 4 - We prepared the technical specifications for the first part of the system module (second iteration). While the first part was being developed, TK was prepared for the second part (third iteration). 

    The first iteration was the preparation of the Conceptual Terms of Reference where the business process was completely written using CIS, and the first version of the architecture of the future module was developed. 

    Step number 5- After the release of the first part of the module and the start of its trial operation, they began to prepare an addition to the TOR for the first part and the TK for the third part (fourth iteration). In addition included those requirements that were missed at the beginning and those that were unsuccessful in terms of use. 

    Result - For everything about everything, it really took us 1.5 years. A working prototype (the first part of the system module) was obtained after 6 months from the project start date. The rest came every 2 months.



    From the point of view of architecture, the components of the new functional module were sharpened exclusively for it and did not use the components of the main IC. It allowed:

    1. To stabilize the entire system when errors occur that are regularly generated by changes to the module, as well as the module from errors of changes made to the general functionality of the IP. 
    2. To carry out work on the development of the existing CIS functionality in parallel with the development of a new module. 

    Although RUP is directly associated and associated with UML notations, it is not necessary to use them to describe business and system usage scenarios, other notations will do. The main thing is that the notations used are understandable for both Business Users who will coordinate-approve the statement, and System Analysts and Developers who will continue to work with the Terms of Reference, translating it into design and technical documentation. To describe the business scenario, we used eEPC and ARIS, as well as the requirements of IDEF0 / 3 notations to set the framework for system scenarios: process input, process output, executor.

    The article “What should be the ToR for corporate IP?” Gives an idea of ​​what the business scenario and system use case look like. In the original RUP manual, it is not recommended to sketch user interfaces when compiling system scenarios, but we considered them necessary because they facilitate understanding of both business and system scenarios by visualizing their elements. This was practically the first prototype of the system, although not clickable. 

    After this project, the written TK became a new template (standard). The project also influenced the entire technological and production process of software development in the company making it rhythmic.

    A bit of theory for understanding the technical side of the RUP methodology and how to apply it


    Each iteration in RUP is a classic Waterfall, containing all 5 stages of work on collecting requirements and their analysis, design, development, testing and delivery. But RUP also introduces such a concept as phases for the project: Requirement Collection and Analysis, Design, Construction, Implementation.



    Now more about this.

    The first iteration, Phase Analysis

    Dedicated to gathering the requirements of future users for the system. Users tell each piece of work (the scenario of using the system by the user), and the Analyst tries to assemble an entire workflow from individual operations and procedures. The goal of this iteration is to enter the business scenario of using the system / future IP-based business process. There can be more than one, so do not focus on having one.

    The Customer Company is the Bank. He decides that he wants to start providing remote cash withdrawal services to his clients using ATMs installed in client offices and metro stations.

    To provide this service, it is necessary to develop and modify the software available at the bank. During the collection of requirements and their analysis, three major procedures were identified in the service business scenario:

    1. Loading cash into an ATM by Bank staff.
    2. Cash withdrawal to the Client at his request from a bank account.
    3. Accounting for the number of downloaded and issued bills.

    In this iteration, there are still no works related to the design of the architecture of the future system, program development, testing and delivery. RUP does not set the conditions that they are necessary, but you can fulfill them at your discretion.

    Second iteration, Phase Analysis

    Once the business scenarios are defined, the collected requirements are laid out according to the steps of the business scenario and system scenarios. At this stage, those steps of business scenarios that were not identified in the first iteration are also revealed.

    During the second iteration, two additional procedures were identified for the main business scenario:

    1. Monitoring and notification sent by the ATM to the bank about the termination of cash in it.
    2. Monitoring and notification of the operability of an ATM or the presence of problems / errors in it.
    To ensure the availability of cash withdrawal services 24/7.

    The result of this iteration is:
    1. 90% complete business scenarios.
    2. Intended, but not detailed, system usage scenarios for each business scenario.
    3. The collected requirements are structured according to business and system scenarios.

    Third iteration, Phase Design

    Identified business scenarios set implementation priorities and the most important of them become the subject of study at this iteration. In the selected business scenarios, the primary steps-stages are highlighted, which are the basis for all subsequent ones and without which the implementation of the rest does not make sense. Scenarios for using the system are detailed for these steps.

    The final result is sent to System Analysts and the Architect to simulate what will happen inside the system. Classes, cooperation between them, and interaction sequences are distinguished. A basic idea of ​​the architecture of the future system is formed, subsystems, system components and their distribution among the nodes of the IT solution are highlighted.

    Already in this iteration, development work can begin and the result of the iteration will be a ready-made prototype that supports the implementation of the primary steps, stages of a key business scenario. It can already be shown to users to collect feedback and clarify requirements.

    This prototype is not transferred to trial or industrial operation, as the underlying architecture is the first test ball and will be adjusted in subsequent iterations.



    Fourth iteration, Phase Design

    Starts parallel to the third iteration, when it completed the analysis work and began work on the design.



    At this iteration, the primary steps-stages of the remaining business scenarios are being worked out. The reason they are being worked out, rather than the next steps for key business scenarios, is to determine the future architecture of the entire program.

    Each business scenario may require for its implementation some specific components of the future system, and for this reason, architectural decisions for each business scenario must be consistent with each other.

    The result of this iteration will be:

    1. Architectural solutions for every business scenario.
    2. Consensus Project 1.0 System Architecture
    3. The second version of the prototype with ready-made first steps of secondary business scenarios in addition to the first steps of a key business scenario.

    Fifth iteration, Build Phase

    Starts parallel to the fourth iteration, when the analysis work is completed and design work is started. It is assumed that by the start of the iteration, feedback has already been received according to the results of testing by users of a prototype made during the third iteration. Here, the remaining steps of the key business scenario and the associated system usage scenarios are worked out.

    It is assumed that by the time the “Design” stage of work begins, the fourth iteration will be implemented (developed) and the second version of the prototype will be tested.
    The agreed-upon project of the overall system architecture is refined, after which the introduction of some conceptual changes to it is practically reduced to zero, but it is still possible, since completely secondary business scenarios are not implemented.

    The result of the iteration is:
    1. A fully implemented key business scenario.
    2. Agreed Design for a General System 2.0 Architecture
    3. The third version of the system that implements a key business scenario and the first steps for secondary business scenarios.

    It is already possible to transfer the system for trial operation to business users since the system is no longer a prototype and is more similar to the one that will be transferred to business users as its final version.

    Sixth iteration, Build Phase

    Starts parallel to the fifth iteration, when the analysis work is completed and design work is started.

    The remaining steps and stages of secondary business scenarios and associated system use cases are registered. It is assumed that by the beginning of this iteration, the second version of the prototype was tested, made during the fourth iteration, and feedback was received from users on working with the primary steps and stages of secondary business scenarios.

    The result of the iteration is:

    1. Fully implemented secondary business scenarios.
    2. The final project of the overall architecture of the system 3.0
    3. The fourth version of the system that implements all business scenarios.

    The full-fledged trial operation of the system by business users and its fitting for industrial use begins.

    Seventh iteration, Implementation Phase

    Some critical changes are made to the implementation of business scenarios identified during the trial operation and not allowing full use of the system in industrial operation.

    Eighth iteration, Implementation phase

    At this iteration, work is underway to improve the usability of the system by users and to optimize the solutions made. Users are primarily interested in the speed and throughput of their sites. The system is being transferred for support. Formal procedures are underway to complete and close the project.

    Subtotal

    The RUP methodology does not impose conditions on the number of iterations and their phase distribution. The concept of phases and iterations was introduced to reduce risks, resource requirements, set the pace of the project and provide the opportunity to make the necessary changes when developing a large information system.

    Schedule of the duration of the project from the time the problem was detected in the IT product


    The result of each iteration refines the plan for the next and gives more understanding about whether we need additional resources for the next iteration, whether additional iterations are needed to complete the phase. A minimal team of several people can start working on a project and its expansion will be justified by the plan for the next iteration.

    The results of the completion of the phase clarify the project plan and economics. Here decisions are made whether to continue the project, whether it is necessary to narrow or expand the content of the project (the volume of business requirements) in order to remain within the business framework: the required delivery time for the project result, its cost (budget) and profitability (return on investment).

    Conclusion


    I understand that it’s not easy to change the principles adopted by the company how to implement IT projects, especially if the company has a Project Office with a methodology written on the basis of PMBoK and GOST. If after reading the article you understand that the immediate prospect is still Waterfall, then I recommend trying to implement the following minimum:

    1. Prescribe an automated business process based on the use of IP. This is more than 50% ready-made user documentation and acceptance tests.

    2. To lay down time for the project that at least 1 time during the year will have to complete a full range of work on the redesign, preparation and approval of changes to the design documentation.

    3. Multiply the complexity of development work by 1.5. Since a third of the work done will have to be thrown out and new requirements will appear. The duration of the development phase should also be multiplied by 1.5.

    4. Divide the project budget into two parts: creation and development.

      For development, lay about 30% of the total project budget for creation. This is necessary then, that when you finish the project an intermediate state will arise - the project is implemented, but not accepted for maintenance .

      Acceptance for escort, as a rule, is accompanied by procedures for coordination and allocation of the budget and resources for escort. At the same time, after the end of the project, the hottest phase begins, when users will demand the implementation of critical improvements that are not included in the content (scope) of the project, but they definitely need to be made and financed from something until the system is accepted for maintenance and transferred to support budget.

      This will allow to gently agree with the Customers about the end of the project implementation phase, carried out in accordance with the statement of work.

    In my opinion:

    • Waterfall is the first gear (first speed with a release once a year),
    • RUP - second (quarterly release),
    • Scrum - third (bi-weekly / monthly release),
    • Kanban along with automatic update building (Continuous Integration), automatic testing (Acceptance Test-Driven Development), automatic delivery and deployment (Continuous Deployment) is the fourth (once-a-day release).

    What else to read on the topic for the development of RUP


    1. Статья «Каким должно быть ТЗ на корпоративную ИС?»

      В статье делается упор на то, что после разработки и создания КИС, начнётся этап её развития. На этом этапе потребуется разработка как новых Технических Заданий на новые функциональные модули, так и внесение изменений в ранее разработанные ТЗ на существующие модули системы в связи с внесением изменений в сценарии их использования.

      Обратите внимание на комментарии под статьёй. Они шире раскрывают материал и самой статьи про ТЗ и этой статьи про RUP.

    2. Книга «Унифицированный процесс разработки программного обеспечения»

      Это настольная книга в которой есть всё что нужно, чтобы понять и научиться делать проекты по RUP.

      Первая и основная сложность — это научиться проектировать не существующие бизнес-процессы / бизнес-сценарии. Чтобы подступится к ним я рекомендую научиться описывать существующие, после чего проектирование нового взамен старого уже не будет таким сложным.

      Авторы книги предлагают для моделирования бизнес-процессов использовать UML, но всё таки книга заточена на проектирование информационной системы и в меньшей степени на проектирование бизнес-процессов. Поэтому рекомендую освоить нотации IDEFx на основе Structured Analysis and Design Technique (книга на русском). The process concept in this methodology is the basis for ISO and PMI standards. And after that, expand your knowledge and skills in using other notations.

    3. Learning a modeling language is directly related to the need to master a tool. RUP expects you to use the Rational product line from IBM. I recommend using the free demo version of Business Studio . This video from 12 minutes explains how to build models using this tool .

    Also popular now: