Multithreaded application for Tornado

The documentation for the non-blocking web server Tornado nicely describes how well it copes with the load, and in general is the crowning achievement of humanity in the field of non-blocking servers. This is partly true. But when building complex applications outside the framework of “another chat”, many unobvious and subtle moments come to light that you would like to know about before going to a rake. Under the “cut”, the developers of the Trelyazh intellectual games club are ready to share their thoughts about the pitfalls.

Immediately make a reservation that we are talking about the second branch of python, the latest version of tornado 1.2.1, and postgresql connected via psycopg2.

Application instance

Many programmers love to use the singleton anti- pattern for quick access
to the application class instance. If we are thinking about further horizontal scaling,
then this is highly discouraged. The request object will bring you a thread-safe
application on a silver platter, which can be used without risking to shoot your foot in the most
unexpected place.

Websockets

Alas and ah. The beloved nginx is not able to proxy the websocket protocol. For fans of "lay", there is also little good news in this regard. Many praise ha-proxy, but in our case it was more convenient to transfer all the statics to another node with honest nginx, and give all the dynamic content to the tornado-server themselves. Six months of life under, sometimes stressful loads, showed that this solution is quite viable. If a flash strip is used to emulate the ws protocol, then it must be sent from the same domain in order to avoid switching to the insecure version. The solution with laying also requires flash policy xml, which can be sent from port 843 to the same nginx.

Connection to the database

Obviously, on loaded services, there can be no talk of terribly expensive connection operations with the database for each sneeze. It is quite possible to use the simplest regular connection pool from psycopg2. Immediately take the thread-safe ThreadedConnectionPool from which, as necessary, select the connection, and after the end of the request do not forget to return it back. By “don’t forget to return” means NEVER forget. Whatever exception we have happened inside. Using the python finally construct is more than appropriate.

Asynchronous requests

In single-threaded non-blocking servers, everything looks beautiful, exactly until you have to perform some kind of relatively long action. Send a letter, make a selection from the database, send a request to an external web service, etc. In this case, all other connected clients will dutifully wait for the handler's turn to reach them.

If all you need in the request handler is just to pull the database and display something, then you can use momoko asynchronous wrapper . Then the simplest query would look something like this:

class MainHandler(BaseHandler):
    @tornado.web.asynchronous
    def get(self):
        self.db.execute('SELECT 4, 8, 15, 16, 23, 42;', callback=self._on_response)
    def _on_response(self, cursor):
        self.write('Query results: %s' % cursor.fetchall())
        self.finish()

Secure multithreading

So, for the base and for external web services there are asynchronous tools. But what if you need to do a large chunk of work from many sql queries, recount something cumbersome in the bowels of the server, and even load the disk i / o subsystem? Of course, we can concoct clusters of asynchronous callback functions in the worst twisted traditions, but this is exactly what we would like to get away from.

Here, it would seem, the use of standard threading suggests itself. But using regular python threads will lead to just monstrous glitches and disastrous results on production under load. Yes Yes. Everything will work fine on the machines of the developers. Usually in such cases, programmers begin to pray on the GIL, and frantically lock everything that is possible and that is impossible. But the problem is that not all of tornado is thread safe. In order to get around this, you need to process the http request in several stages.

1. Decode your get / post function with tornado .web.asynchronous
2. Accept the request, check the input parameters, if any, and save them in the request instance
3. Run thread from member function of request class
4. Perform all the work inside this function by carefully applying locks at the time the shared data changes
5. Call a callback that will make the final _finish () to the process with the data already prepared.

For these purposes, you can write a small Mixin:

class ThreadableMixin:
    def start_worker(self):
        threading.Thread(target=self.worker).start()
    def worker(self):
        try:
            self._worker()
        except tornado.web.HTTPError, e:
            self.set_status(e.status_code)
        except:
            logging.error("_worker problem", exc_info=True)
            self.set_status(500)
        tornado.ioloop.IOLoop.instance().add_callback(self.async_callback(self.results))
    def results(self):
        if self.get_status()!=200:
            self.send_error(self.get_status())
            return
        if hasattr(self, 'res'):
            self.finish(self.res)
            return
        if hasattr(self, 'redir'):
            self.redirect(self.redir)
            return
        self.send_error(500)

And in this case, secure multi-threaded request processing will look simple and elegant:

class Handler(tornado.web.RequestHandler, ThreadableMixin):
    def _worker(self):
        self.res = self.render_string("template.html",
            title = _("Title"),
			data = self.application.db.query("select ... where object_id=%s", self.object_id)
        )
    @tornado.web.asynchronous
    def get(self, object_id):
		self.object_id = object_id
        self.start_worker()

If we need a redirect, then in _worker () we set the self.redir variable to the desired url. If you need json for an ajax request, then in self.res, instead of the generated page, we assign the generated dict with the data.

Another point is related to C-extensions for python. If you use any external libraries in call flows, be sure to check their thread-safe status.

Batch processes

Often you need to run a function after a strictly defined time. These are user timeouts, and the implementation of game processes, and system maintenance procedures, and much more. For these purposes, the so-called "periodic processes" are used.

Traditionally, to organize batch processes, python uses standard threading.Timer. If we try to use it, then again we get a certain amount of elusive problems. For these purposes, tornado provides ioloop.PeriodicCallback. Please always use it instead of regular timers. This will save a lot of time and nerves for the above reasons.

Localization and more

In conclusion, let me give you some tips that are not related to multi-threaded processing, but sometimes allow you to significantly increase the performance of a branchy application.

1. Do not use the built-in tornado stub for localization. Tornado perfectly knows how to bind to the standard gettext and gives it much better results on large volumes of translations.
2. Cache in memory all that is possible. Forget memcached & co. You do not need it. Already in the design process, you should know on which hardware platform your application will run. The extra couple of gigabytes of memory in the server can fundamentally change the approach to a particular caching strategy.
3. If the page generation time entirely depends on the data in the system and you cannot know its limits in advance, always postpone a new thread for this request
4. Despite the fact that the Tornado is very fast, always give static to the means intended for this. For example nginx. You just can’t imagine what the i7 / 16Gb / SAS server with FreeBSD / amd64 and nginx on board is capable of on handing out statics. Nothing can be faster just physically.

Result

During stress testing, 5000 simultaneous connections that are actively playing on the site (which means thousands of websocket messages per second) the server pulls without problems (LA ~ = 0.2, the server process eats up about 400Mb of memory with 8Gb free). 150 real players online, fun writing bullets, the server does not notice at all (zero load and a huge supply of power).

At the front, it looks something like this: And may the force be with you!