Simple and fast application stress testing framework

[ english version ]

Recently, more and more web applications for high loads have been created, but with frameworks allowing them to flexibly test their stress and add their own logic - not a lot.
Of course, there are a lot of different ones (see the vote at the end of the post), but someone does not support cookies, someone gives a weak load, someone is very heavy, and they are mostly suitable for very similar requests, i.e. dynamically generate each request using its own logic and at the same time as fast as possible (and ideally in java to finish if that) - I did not find such.

Therefore, it was decided to sketch my own, because these are only 3-5 classics in this case. Basic requirements: speed and dynamic query generation. At the same time, speed is not just thousands of RPS, but ideally - when stress depends only on network bandwidth and works from any free machine.

Engine

With the requirements it is clear, now you need to decide on what it will all work on, i.e. which http / tcp client to use. Of course, we don’t want to use the outdated thread-per-connection model, because we immediately run into several thousand rps depending on the power of the machine and the speed of context switching in jvm. T.O. apache-http-client and the like are swept away. Here you need to look at the so-called non-blocking network clients built on NIO .

Fortunately, in the java world, in this niche there has long been a standard de facto open source Netty , which is also very versatile and low-level, allowing you to work with tcp and udp.

Architecture

To create our sender, we need a ChannelUpstreamHandler handler in terms of Netty, from which our requests will be sent.

Next, you need to select a high-performance timer to send the maximum possible number of requests per second (rps). Here you can take the standard ScheduledExecutorService , it basically copes with this, but on weak machines it is better to use HashedWheelTimer (included in Netty) due to lower overhead when adding tasks, it only requires some tuning. On powerful machines, there is practically no difference between them.

And the last, in order to squeeze the maximum rps from any machine when it is not known what limits on connections in this OS or the total current load, it is most reliable to use the trial and error method: first set some transcendental value, for example, a million requests per second and then wait on how many connections will errors begin when creating new ones. Experiments have shown that the maximum number of rps is usually slightly less than this figure.
Those. we take this figure for the initial rps value and then if the errors are repeated, we reduce it by 10-20%.

Implementation

Request Generation

To support dynamic query generation, we create an interface with the only method that our stress will cause to receive the contents of the next request:

public interface RequestSource {
    /**
     * @return request contents
     */
    ChannelBuffer next();
}

ChannelBuffer is an abstraction of byte stream in Netty, i.e. here, the entire contents of the request should be returned as a stream of bytes. In the case of http and other text protocols, this is just a byte representation of the query string (text).
Also, in the case of http, it is necessary to put 2 newlines at the end of the request (\ n \ n), this is a sign of the end of the request for Netty (will not send the request otherwise)

Dispatch

To send requests to Netty, you first need to explicitly connect to the remote server, so at the start of the client we start periodic connections with a frequency in accordance with the current rps:

scheduler.startAtFixedRate(new Runnable() {
    @Overrid
    public void run() {
       try {
            ChannelFuture future = bootstrap.connect(addr);
            connected.incrementAndGet();
        } catch (ChannelException e) {
            if (e.getCause() instanceof SocketException) {
                processLimitErrors();            
            }
            ...
        }, rpsRate);

After a successful connection, we immediately send the request itself, so our Netty handler will conveniently inherit from SimpleChannelUpstreamHandler where there is a special method for this. But there is one caveat: a new connection is processed by the so-called the main thread (“boss”), where long operations should not be present, which may be the generation of a new request, so you have to shift it to another thread, as a result, sending the request itself will look something like this:

private class StressClientHandler extends SimpleChannelUpstreamHandler {        
        ....
        @Override
        public void channelConnected(ChannelHandlerContext ctx, final ChannelStateEvent e) throws Exception {
            ...
            requestExecutor.execute(new Runnable() {
                @Override
                public void run() {
                    e.getChannel().write(requestSource.next());
                }
            });
            ....
        }
    }

Error processing

Next is error handling for creating new connections when the current frequency of sending requests is too high. And this is the most non-trivial part, or rather it’s difficult to do it platform independent, because different operating systems behave differently in this situation. For example, linux throws a BindException, windows throws a ConnectException, and MacOS X throws either one of these, or an InternalError (Too many open files) in general. T.O. on the poppy axis, stress behaves most unpredictably.

In this regard, in addition to handling errors during connection, it is also necessary to do this in our handler (simultaneously counting the number of errors for statistics):

private class StressClientHandler extends SimpleChannelUpstreamHandler {        
        ....
        @Override
        public void exceptionCaught(ChannelHandlerContext ctx, ExceptionEvent e) throws Exception {
            e.getChannel().close();
            Throwable exc = e.getCause();
            ...
            if (exc instanceof BindException) {
                be.incrementAndGet();
                processLimitErrors();
            } else if (exc instanceof ConnectException) {
                ce.incrementAndGet();
                processLimitErrors();
            } 
            ...
        }
            ....
   }

Server responses

Finally, we need to decide what we will do with the responses from the server. Since this is a stress test and only bandwidth is important to us, it remains only to read the statistics:

private class StressClientHandler extends SimpleChannelUpstreamHandler {
  @Override
        public void messageReceived(ChannelHandlerContext ctx, MessageEvent e) throws Exception {
            ...
            ChannelBuffer resp = (ChannelBuffer) e.getMessage();
            received.incrementAndGet();
            ...
        }
}

There may also be a calculation of the types of http responses (4xx, 2xx)

Whole code

All code with additional goodies like reading http templates from files, template engine, timeouts, etc. lies in the form of a finished maven project on GitHub (ultimate-stress) . There you can download the finished distribution kit (jar file).

conclusions

All of course rests against the limit of open connections. For example, on linux, while increasing some OS settings (ulimit, etc.), on the local machine it was possible to achieve about 30K rps, on modern hardware. Theoretically, in addition to the limit of connections and the network, there should not be more restrictions, in practice, however, the overhead jvm makes itself felt and the actual rps is 20-30% less than the specified one.