You will answer for everything! Consumer Driven Contracts through the eyes of the developer

    In this article, we will talk about the problems that Consumer Driven Contracts solves, and show how to apply it using the example of Pact with Node.js and Spring Boot. And talk about the limitations of this approach.


    Issue


    When testing products, scenario tests are often used in which the integration of various components of the system in a specially selected environment is checked. Such tests on live services give the most reliable result (not counting tests in battle). But at the same time, they are one of the most expensive.

    • It is often mistakenly believed that the integration environment should not be fault tolerant. SLA, guarantees for such environments are rarely spoken out, but if it is not available, teams have to either delay releases, or hope for the best and go into battle without tests. Although everyone knows that hope is not a strategy . And newfangled infrastructure technologies only complicate work with integration environments.
    • Another pain is working with test data . Many scenarios require a certain state of the system, fixtures. How close should they be to combat data? How to bring them up to date before the test and clean after completion?
    • Tests are too unstable . And not only because of the infrastructure that we mentioned in the first paragraph. The test may fail because a neighboring team launched its own checks that broke the expected state of the system! Many false negative checks, flaky tests end their lives in @Ignored. Also, different parts of the integration may be supported by different teams. They rolled out a new release candidate with errors - they broke all consumers. Someone solves this problem with dedicated test loops. But at the cost of multiplying the cost of support.
    • Such tests take a lot of time . Even with automation in mind, results can be expected for hours.
    • And to top it off, if the test really fell right, it is far from always possible to immediately find the cause of the problem. It can hide deep behind layers of integration. Or it may be the result of an unexpected combination of states of many system components.

    Stable tests in an integration environment require serious investment from QA, dev and even ops. No wonder they are at the very top of the test pyramid . Such tests are useful, but the resource economy does not allow them to check everything. The main source of their value is the environment.

    Below the same pyramid are other tests in which we exchange confidence for smaller support headaches - using isolation checks. The granular, the smaller the scale of the test, the less the dependence on the external environment. At the very bottom of the pyramid are unit tests. We check individual functions, classes, we operate not so much with business semantics as with constructions of a specific implementation. These tests give quick feedback.

    But as soon as we go down the pyramid, we have to replace the environment with something. Stubs appear - as whole services, and individual entities of the programming language. It is with the help of plugs that we can test components in isolation. But they also reduce the validity of the checks. How to make sure that the stub is returning the correct data? How to ensure its quality?

    The solution can be comprehensive documentation that describes various scenarios and possible states of system components. But any formulations still leave freedom of interpretation. Therefore, good documentation is a living artifact that is constantly improving as the team understands the problem area. How then to ensure compliance with documentation stubs?

    On many projects, you can observe a situation where the stubs are written by the same guys who developed the test artifact. For example, developers of mobile applications make stubs for their tests themselves. As a result, programmers can understand the documentation in their own way (which is completely normal), they make the stub with the wrong expected behavior, write the code in accordance with it (with green tests), and errors occur during real integration.

    Moreover, the documentation usually moves downstream - clients use specs of services (in this case, another service can be a client of the service). It does not express how consumers use data, what data is needed at all, what assumptions they make for that data. The consequence of this ignorance is the law of Hyrum .



    Hyrum Wright has been developing public tools inside Google for a long time and has observed how the smallest changes can cause breakdowns for customers who used the implicit (undocumented) features of his libraries. Such hidden connectivity complicates the evolution of the API.

    These problems can be resolved to some extent using Consumer Driven Contracts. Like any approach and tool, it has a range of applicability and cost, which we will also consider. Implementations of this approach have reached a sufficient level of maturity to try on their projects.

    What is a CDC?


    Three key elements:

    • The contract . Described using some DSL, implementation dependent. It contains a description of the API in the form of interaction scenarios: if a specific request arrives, then the client should receive a specific response.
    • Customer tests . Moreover, they use a stub, which is automatically generated from the contract.
    • Tests for the API . They are also generated from the contract.

    Thus, the contract is executable. And the main feature of the approach is that the requirements for the behavior of the API go upstream , from the client to the server.

    The contract focuses on the behavior that really matters to the consumer. Makes its assumptions about the API explicit.

    The main objective of the CDC is to bring an understanding of the behavior of the API to its developers and the developers of its clients. This approach is well combined with BDD, at meetings of three amigo you can sketch the blanks for the contract. Ultimately, this contract also serves to improve communications; sharing a common understanding of the problem area and implementing the solution within and between teams.

    Pact


    Consider using CDC as an example from Pact, one of its implementations. Suppose we make a web application for conference participants. In the next iteration, the team develops a presentation schedule - so far without any stories like voting or notes, only the output of the reports grid. The source code for the example is here .

    At a meeting of three four amigo, a product, a tester, developers of the backend and a mobile application meet. They say that

    • A list with the text will be displayed in the UI: Report title + Speakers + Date and time.
    • To do this, the backend must return data as in the example below.

    {
       "talks":[
          {
             "title":"Изготовление качественных шерстяных изделий в условиях невесомости",
             "speakers":[
                {
                   "name":"Фубар Базов"
                }
             ],
             "time":"2019-05-27T12:00:00+03:00"
          }
       ]
    }

    After which the frontend developer goes to write the client code (backend for frontend). He installs a pact contract library in the project:

    yarn add --dev @pact-foundation/pact

    And begins to write a test. It configures the local stub server, which will simulate the service with report schedules:

    const provider = new Pact({
      // название потребителя и поставщика данных
      consumer: "schedule-consumer",
      provider: "schedule-producer",
      // порт, на котором поднимется заглушка
      port: pactServerPort,
      // сюда pact будет писать отладочную информацию
      log: path.resolve(process.cwd(), "logs", "pact.log"),
      // директория, в которой сформируется контракт
      dir: path.resolve(process.cwd(), "pacts"),
      logLevel: "WARN",
      // версия DSL контракта
      spec: 2
    });

    The contract is a JSON file that describes the scenarios of the client interacting with the service. But you do not need to manually describe it, since it is formed from the settings of the stub in the code. The developer before the test describes the following behavior.

    provider.setup().then(() =>
      provider
        .addInteraction({
          uponReceiving: "a request for schedule",
          withRequest: {
            method: "GET",
            path: "/schedule"
          },
          willRespondWith: {
            status: 200,
            headers: {
              "Content-Type": "application/json;charset=UTF-8"
            },
            body: {
              talks: [
                {
                  title: "Изготовление качественных шерстяных изделий в условиях невесомости",
                  speakers: [
                    {
                      name: "Фубар Базов"
                    }
                  ],
                  time: "2019-05-27T12:00:00+03:00"
                }
              ]
            }
          }
        })
        .then(() => done())
    );

    Here, in the example, we specified the specific expected service request, but pact-js also supports several methods for determining matches .

    Finally, the programmer writes a test of that part of the code that uses this stub. In the following example, we will call it directly for simplicity.

    it("fetches schedule", done => {
      fetch(`http://localhost:${pactServerPort}/schedule`)
        .then(response => response.json())
        .then(json => expect(json).toStrictEqual({
          talks: [
            {
              title: "Изготовление качественных шерстяных изделий в условиях невесомости",
              speakers: [
                {
                  name: "Фубар Базов"
                }
              ],
              time: "2019-05-27T12:00:00+03:00"
            }
          ]
        }))
        .then(() => done());
    });

    In a real project, this can be either a quick unit test of a separate response interpretation function or a slow UI test for displaying data received from a service.

    During the test run, pact verifies that the stub received the request specified in the tests. The discrepancies can be seen as diff in the pact.log file.

    E, [2019-05-21T01:01:55.810194 #78394] ERROR -- : Diff with interaction: "a request for schedule"
    Diff
    --------------------------------------
    Key: - is expected 
         + is actual 
    Matching keys and values are not shown
     {
       "headers": {
    -    "Accept": "application/json"
    +    "Accept": "*/*"
       }
     }
    Description of differences
    --------------------------------------
    * Expected "application/json" but got "*/*" at $.headers.Accept


    If the test succeeds, a contract is generated in JSON format. It describes the expected behavior of the API.

    {
      "consumer": {
        "name": "schedule-consumer"
      },
      "provider": {
        "name": "schedule-producer"
      },
      "interactions": [
        {
          "description": "a request for schedule",
          "request": {
            "method": "GET",
            "path": "/schedule",
            "headers": {
              "Accept": "application/json"
            }
          },
          "response": {
            "status": 200,
            "headers": {
              "Content-Type": "application/json;charset=UTF-8"
            },
            "body": {
             "talks":[
                {
                   "title":"Изготовление качественных шерстяных изделий в условиях невесомости",
                   "speakers":[
                      {
                         "name":"Фубар Базов"
                      }
                   ],
                   "time":"2019-05-27T12:00:00+03:00"
                }
             ]
           }}}
      ],
      "metadata": {
        "pactSpecification": {
          "version": "2.0.0"
        }
      }
    }

    He gives this contract to the backend developer. Let's say the API is on Spring Boot. Pact has a pact-jvm-provider-spring library that can work with MockMVC. But we'll take a look at the Spring Cloud Contract, which implements CDC in the Spring ecosystem. It uses its own format of contracts, but also has an extension point for connecting converters from other formats. Its native contract format is supported only by the Spring Cloud Contract itself - unlike Pact, which has libraries for JVM, Ruby, JS, Go, Python, etc.

    Suppose, in our example, the backend developer uses Gradle to build the service. It connects the following dependencies:

    buildscript {
    	// ...
    	dependencies {
    		classpath "org.springframework.cloud:spring-cloud-contract-pact:2.1.1.RELEASE"
    	}
    }
    plugins {
    	id "org.springframework.cloud.contract" version "2.1.1.RELEASE"
           // ...
    }
    // ...
    dependencies {
        // ...
        testImplementation 'org.springframework.cloud:spring-cloud-starter-contract-verifier'
    }

    And it puts the Pact contract received from the frotender into the directory src/test/resources/contracts.

    From it, by default, the spring-cloud-contract plugin subtracts contracts. During assembly, the generateContractTests gradle task is executed, which generates the following test in the build / generated-test-sources directory.

    public class ContractVerifierTest extends ContractsBaseTest {
        @Test
        public void validate_aggregator_client_aggregator_service() throws Exception {
            // given:
            MockMvcRequestSpecification request = given()
                .header("Accept", "application/json");
            // when:
            ResponseOptions response = given().spec(request)
                .get("/scheduler");
            // then:
            assertThat(response.statusCode()).isEqualTo(200);
            assertThat(response.header("Content-Type")).isEqualTo("application/json;charset=UTF-8");
            // and:
            DocumentContext parsedJson = JsonPath.parse(response.getBody().asString());
            assertThatJson(parsedJson).array("['talks']").array("['speakers']").contains("['name']").isEqualTo( /*...*/ );
            assertThatJson(parsedJson).array("['talks']").contains("['time']").isEqualTo( /*...*/ );
            assertThatJson(parsedJson).array("['talks']").contains("['title']").isEqualTo( /*...*/ );
        }
    }


    When starting this test, we will see an error:

    java.lang.IllegalStateException: You haven't configured a MockMVC instance. You can do this statically

    Since we can use different tools for testing, we need to tell the plug-in which one we have configured. This is done through the base class, which will inherit the tests generated from the contracts.

    public abstract class ContractsBaseTest {
        private ScheduleController scheduleController = new ScheduleController();
        @Before
        public void setup() {
            RestAssuredMockMvc.standaloneSetup(scheduleController);
        }
    }


    To use this base class during generation, you need to configure the spring-cloud-contract gradle plugin.

    contracts {
    	baseClassForTests = 'ru.example.schedule.ContractsBaseTest'
    }


    Now we have the following test generated:
    public class ContractVerifierTest extends ContractsBaseTest {
    	@Test
    	public void validate_aggregator_client_aggregator_service() throws Exception {
    		// ...
    	}
    }

    The test starts successfully, but fails with a verification error - the developer has not yet written the implementation of the service. But now he can do it based on a contract. He can make sure that he is able to process the client’s request and return the expected response.

    The service developer knows through the contract what he needs to do, what behavior to implement.

    Pact can be integrated deeper into the development process. You can deploy a Pact-broker that aggregates such contracts, supports their versioning, and can display a dependency graph.



    Uploading a new generated contract to the broker can be done in step CI when building the client. And in the server code indicate the dynamic loading of the contract by URL. Spring Cloud Contract also supports this.

    CDC Applicability


    What are the limitations of Consumer Driven Contracts?

    For using this approach you have to pay with additional tools like pact. Contracts per se are an additional artifact, another abstraction that must be carefully maintained and consciously applied engineering practices to it.

    They do not replace e2e tests , since stubs still remain stubs - models of real system components, which may be a little bit, but do not correspond to reality. Through them, complex scenarios cannot be verified.

    Also, CDCs do not replace API functional tests.. They are more expensive to support than Plain Old Unit Tests. Pact developers recommend using the following heuristics - if you remove the contract and this does not cause errors or misinterpretation by the client, then it is not needed. For example, it is not necessary to describe absolutely all API error codes through a contract if the client processes them the same way. In other words, the contract describes for the service only what is important to its client . No more, but no less.

    Too many contracts also complicate the evolution of the API. Each additional contract is an occasion for red tests.. It is necessary to design a CDC in such a way that each fail test carries a useful semantic load that outweighs the cost of its support. For example, if the contract fixes the minimum length of a certain text field that is indifferent to the consumer (he uses the Toleran Reader technique ), then every change to this minimum value will break the contract and the nerves of those around him. Such a check needs to be transferred to the level of the API itself and implemented depending on the source of restrictions.

    Conclusion


    CDC improves product quality by explicitly describing integration behavior. It helps customers and service developers to reach a common understanding, allows you to talk through code. But this does at the cost of adding tools, introducing new abstractions and additional actions of team members.

    At the same time, CDC tools and frameworks are being actively developed and have already reached maturity for testing on your projects. Test :)

    At the QualityConf conference on May 27-28, Andrei Markelov will talk about testing techniques on prod, and Arthur Khineltsev will talk about monitoring a highly loaded front-end, when the price of even a small error is tens of thousands of sad users.

    Come chat for quality!

    Also popular now: