Testing in Yandex. Web-service over SSH itself, or how to make a stub for an entire service

    You are a practicing magician manager. Or a combat developer. Or a professional tester. Or maybe just a person who is not indifferent to the development and use of systems that include client-server components. I am sure you even know that the port is not only the place where the ships come, and “ssh” is not only the sound made by the snake. And you are aware that services located on one or several machines actively communicate with each other. Most often over HTTP. And from version to version, the format of this communication needs to be controlled.

    I think that each of you at the next release asked yourself the following questions: “Are we sending the right request exactly?” Or “Are we passing all the necessary parameters to this service exactly?” Everyone should be aware of the existence of negative scenarios along with positive ones. This knowledge should actively raise questions from the series “What if ..?”. What if the service starts processing connections with a delay of 2 hours? What if the service answers abracadabra instead of data in json format?

    Such things are often forgotten in the development process. Because of the complexity of checking problems of this kind, the unlikelyness of such situations and for a thousand other reasons. But a strange error or application crash at a crucial moment can frighten the user away forever, and he will no longer return to your product. We at Yandex constantly keep such questions in mind and strive to optimize the testing process as much as possible, using useful ideas. How we made such checks easy, visual, automatic will be discussed in this article.


    There are a number of long-known ways to find out how and what is transferred from service to service - from a mobile application to a server, from one part to another.
    The first of the most popular and not requiring serious preparation is to connect a special program to one of the sources of data transmission or reception. Such programs are called traffic analyzers or more often sniffers.
    The second is to replace entirely one of the parties with artificial implementation. In this approach, it is possible to determine a clear scenario of behavior in certain cases and save all the information that will come to this service. This approach is called using stubs ( mock objects ). We will look at both.

    Using sniffers

    A new release or debugging of changes affecting interservice communication comes. We are armed with the necessary interceptors - WireShark or Tcpdump . We start intercepting traffic to the desired node by applying filters on the host, port and interface. We do "dark things", initiating the communication we need. Stop the interception. We begin to disassemble it. The process of parsing each service has its own, but usually always resembles frantic searches in the heap of the text of the treasured GET, POST, PUT, etc. Have you found? Then we repeat this from release to release. This is now a regression test! Did not find? We repeat this with different combinations of filters until we understand the reasons.

    From release to release?

    You can do this manually once. Or two. Well maybe three. And then she’ll get tired of it. And on the fifth release, this communication will take and break. It is especially difficult to notice this indirectly when communication is a callback call with some kind of notification. Repeated from release to release, mechanical actions that take a lot of time and effort are worth automating. How to do it in JAVA? I’m sure that in other languages ​​this can be done in a similar way, but specifically the JUnit4 + Maven bundle for Yandex test automation works fine and has proven itself well.

    Suppose that we are testing the service integratively, which means that most likely it looks like a combat mode when it is raised on a separate machine, and we are connected to it via SSH. We take the library to work on SSH ,We connect to the server , run tcpdump and catch everything we can into the file (everything is exactly like hands). After the test, we force the process to end and look for what we need in the file using grep, awk, sed, etc. Then the resulting process in the test. Did you do that? Not? And no need!

    Why not do this?

    I want to note that “not necessary” does not mean “not possible”.

    You can do that. There are simply ways easier, because:
    • Parsing into components of large lines is always a lot of specific code that is difficult to maintain.
    • Parsing HTTP messages has long been done in hundreds of libraries. Why do one hundred and first?

    We tried exactly this method and we at Yandex. At first it seemed convenient - we only had to start and stop tcpdump via ssh, and then find the necessary substring in its output. The first problems with support began almost immediately: the order of the query parameters turned out to be random in the requested queries. I had to break the reference line into several and check each entry. The error messages were also depressing if the request was not found - tons of text did not provide an adequate way to structure yourself. Asynchronous requests added to the headache, which could appear a few minutes after the action. I had to build dizzying constructions in anticipation of the required substring in the output with a certain delay. The test code sometimes became more complex than the code we tested.

    Using stubs instead of web services

    Since we are talking about testing, then most likely we have all the possibilities not only to put the grid in the form of a sniffer, but also to replace one of the services as a whole. With this approach, “clean” requests will reach the artificial service, and we will have the opportunity to control the behavior of the endpoint for messages. For such purposes, there is a wonderful WireMock library . Its code can be viewed on the project’s GitHub page . The bottom line is that the web service comes up with a good REST-api, which can be configured in almost any way. This is a JAVA library, but it has the ability to run as a standalone application, jre would be available. And then a simple setup with detailed documentation.

    There are arbitrary response codes, and arbitrary content, and transparent request redirection to real services with the ability to save answers and then send them yourself. Particularly noteworthy is the opportunity to recreate negative behavior: timeouts, disconnections, invalid answers. Beauty! The library can also work as a WAR, which can be downloaded into Jetty, Tomcat, etc. And, most importantly, this library can be used directly in tests like JUnit Rule! She herself will take care of parsing the request by dividing the body, address, parameters and headers. All that remains for us is to get a list of all those who came and meet the criteria at the right time.

    Automate checks using a stub

    Before continuing, you need to decide what steps you need to go through to ensure easy, clear and automatic testing using the second of the options considered - a mock-object.
    It is worth noting that each of the stages is also possible to do manually without much difficulty.


    What are we checking? {сообщение}in the diagram:
    тестируемый_сервис -> {сообщение} -> сервис_заглушка.

    More precisely, the diagram will look like this:
    тестируемый_сервис -> сервис_заглушка :(его лог): {сообщение}.

    Thus, we need to do a few things:
    • Raise the service stub and force it to accept certain messages, responding OK (or not OK - depending on the scenario). WireMock will do this.
    • Ensure delivery of messages to the service stub (in the scheme this ->). We will talk about this stage separately.
    • To validate what has come. There are two options - using WireMock tools for validation, or getting a list of requests from it, applying matchers to them .

    Raise the artificial web service

    How to manually raise a service is described in detail on the wiremock website in the Running standalone section . How to use in JUnit is also described though. But we will need this in the future, so I will give a little code.

    We create a JUnit rule that will raise the service on the desired port at the start of the test and end after the end:

    public WireMockRule wiremock = new WireMockRule(LOCAL_MOCKED_PORT);

    The beginning of the test will look something like this:

    public void shouldSend3Callbacks() throws Exception {
        // Пусть наша заглушка принимает любые сообщения

    Here we configure the raised web-service so that it responds to any requested address with code 200 with the body "OK". After some simple configuration steps, there are several scenarios. First, we have no problems accessing any port from the client to the machine on which the test is running. In this case, we simply perform the necessary actions within the test case, after which we proceed to validation. The second - we have access only via ssh. However, the ports are covered by a firewall. Here ssh port forwarding (or ssh-tunneling) comes to the rescue. This is discussed below.

    Shorten the way for packages

    We need REMOTE (which with the -R switch) and, accordingly, ssh-access to the machine. This will allow the test service to access its local port, and for us to listen to our own. And everything will work.

    In a nutshell, then ssh port forwarding (or ssh-tunneling) is to pipe through an ssh connection from the port on the remote machine to the port on the local one. Good instructions for use can be found at www.debianadmin.com

    Since we are automating this process, we will consider in detail how to make using this mechanism convenient in tests. Let's start from the very top level - the junit rule interface. It will allow you to throw the connection хост_удаленной_машины:порт -> ssh -> хост_машины_где_фейк_сервис:его_портbefore the start of the test and close the tunnel after it is completed.

    Making a port forwarding junit rule

    Remember the Ganymed SSH2 library. Connect it using maven:


    (The release version can always be seen in Maven Central .)

    We open an example using this library to raise the tunnel through ssh. We understand that we need four parameters. We assume that the test person is "talking" through his local port, so we хост_удаленной_машиныequate to
    There are three parameters left to specify:

    public SshRemotePortForwardingRule forward = onRemoteHost(props().serviceURI())

    Here .forwardToLocal()it is:

    public SshRemotePortForwardingRule forwardToLocal() {
        try {
            hostToForward = InetAddress.getLocalHost().getHostAddress();
        } catch (UnknownHostException e) {
            throw new RuntimeException("Can't get localhost address", e);
        return this;

    It is convenient to make a Junit rule as an heir ExternalResource, redefining it before()for authorization and raising the tunnel, and after()for closing the tunnel and connection.

    The connection itself should look something like this:

    logger.info(format("Try to create port forwarding: `ssh %s -l %s -f -N -R %s:%s:%s:%s`",
                    connection.getHostname(), SSH_LOGIN,
                    hostOnRemote, portOnRemote, hostToForward, portToForward
    connection.requestRemotePortForwarding(hostOnRemote, portOnRemote, hostToForward, portToForward);


    Having successfully caught requests with the muffled service, all that remains is to check them. The easiest way is to use the built-in WireMock tools:
    // Обращаясь к логу искусственного сервиса, убеждаемся в наличии нужных сообщений
    verify(3, postRequestedFor(urlMatching(".*callback.*"))

    A much more flexible way is to simply get a list of the necessary queries, and then, having got certain parameters, apply checks to them:

    List all = findAll(allRequests());
    assertThat("Должны найти хотя бы 1 запрос в логе", all, hasSize(greaterThan(0)));
    assertThat("Тело первого запроса должно содержать определенную строку",
                    all.get(0).getBodyAsString(), containsString("Wow, it's callback!"));

    How it worked in Yandex

    All of the above is a serious general approach. It can be used in many places, either in whole or in part. Now the use of stubs at the integration level works fine in a number of large projects to replace various functions of services. For example, in Yandex we have a file upload service that records information about files not independently, but through another service. They started to download the file - they sent a request. Loaded, counted checksums - another request. We checked the file for viruses, and are ready to continue working with the file - one more. Each next stage continues depending on the response to the previous ones, while the number of connections between services is limited.

    How to verify that the requests really go away and contain all the information about the file in the correct format? How to check what will happen if the request was accepted, but there was no response? First, we check the positive scenario of the development of events - we replace the service that writes to the database with an artificial one, we receive and analyze traffic. (The code examples above are a copy of what happens in the tests.) The tunnel via ssh was required so that the autotests, without possessing superuser rights, could bind to a specific port on the local machine, whose address is always arbitrary, and you could specify in the download service the point of sending requests is your local address and port on an ongoing basis.

    Having successfully tested the positive scenario, it was not difficult for us to add checks for the negative. Just by increasing the response delay time in WireMock to a value greater than the waiting time in the file download service, it was possible to initiate several attempts to send the request.

    //Глушим сервис записи в БД, установив таймаут на ответ в 61 секунду 
    //(больше чем ожидание ответа на 1с)
                    .withFixedDelay((int) SECONDS.toMillis(61))

    After checking that in 120 seconds, while waiting for a response on the service in 60 seconds, two requests arrived, we made sure that the file download service did not hang at the crucial moment.

    waitFor(120, SECONDS);
    verify(2, postRequestedFor(urlMatching(".*service/callback.*"))

    So, the developers envisioned such a development of events, and in this place in this situation, information about the download will not be lost. Similarly, a bug was found on one of the services. It consisted in the fact that if the service was not answered immediately, then the connection remained open for several hours until it was forcibly closed by external monitoring services. This could lead to the fact that in case of network problems in a short time, the connection limit could be completely exhausted and other clients would have to wait in line for several hours. Good thing we checked this before!

    What else is worth saying

    There are a number of limitations to this approach:
    • It will require ssh access to the machine.
    • Port forwarding must be enabled on this machine.
    • You will need to stop the services if you need to take their port and replace it with a stub. And this means that the user needs the right to stop services without a password. This also applies to ports with numbers up to 1024.
    • In some organizations, you cannot forward a port without the permission of administrators.

    Local port forwarding

    In addition to the remote, there is also LOCAL (local), with the key -L. It allows the mirror described above, referring to some port on your local machine, to get to the internal port of the remote machine, hidden behind the firewall. This approach can be an alternative in tests using ssh to enter the server under test and calling curl, wget.


    In tests, in addition to WireMock, analogs may be interesting: github.com/jadler-mocking/jadler or github.com/robfletcher/betamax .

    Also popular now: