Django ORM, gevent and rake in the green

So many choose Django for its simplicity. The code on Django is simple and concise, we think less about crutches and more about business logic.

Gevent is also chosen due to the fact that it is simple, very smart and does not carry a callback hell.

A great idea arises in my head to combine two simple and convenient things together. We patch Django and rejoice in simplicity, conciseness and performance, make a lot of requests to other sites, create subprocesses, generally use our new asynchronous Django to the maximum.
But combining them, we quietly set a few rakes in our path.

Django ORM and DB Connection Pool

Django was created as a framework for synchronous applications. Each process receives a request from the user, processes it completely, sends the result, and only after that it can start processing another request. The principle of operation is simple, like a steamed turnip. Great architecture to create a blog or news site, but we want more.

Take a simple example that simulates a hectic activity with HTTP requests. Let it be a service for shortening links:

# testproject/__init__.py
__import__('gevent.monkey').monkey.patch_all()
# testproject/main/models.py
from django.db import models
class LinkModel(models.Model):
    url = models.URLField(max_length=256, db_index=True, unique=True)
# testproject/main/views.py
import urllib2, httplib
from django.core.urlresolvers import reverse
from django.http import HttpResponse, HttpResponseRedirect
from .models import LinkModel
def check_url(url):
    request = urllib2.Request(url)
    request.get_method = lambda: 'HEAD'
    try:
        response = urllib2.urlopen(request)
    except (urllib2.URLError, httplib.HTTPException):
        return False
    response.close()
    return True
def remember(request):
    url = request.GET['url']
    try:
        link = LinkModel.objects.get(url=url)
    except LinkModel.DoesNotExist:
        if not check_url(url):
            return HttpResponse('Oops :(')
        link = LinkModel.objects.create(url=url)
    return HttpResponse('http://localhost:8000' + reverse(
            go_to, args=(str(link.id).encode('base64').strip(), )))
def go_to(request, code):
    obj = LinkModel.objects.get(id=code.decode('base64'))
    return HttpResponseRedirect(obj.url)

Without gevent, this code would have worked incredibly slowly and would have difficulty serving two or three simultaneous requests, but with gevent everything flies.

We launch our project through uwsgi (which de facto became the standard when deploying Python sites:

uwsgi --http-socket 0.0.0.0:8000 --gevent 1000 -M -p 2 -w testproject.wsgi

We try to test ten requests for shortening the link at the same time and rejoice: all requests fulfill without errors in a minimum amount of time.

We launch our new service, sit and look at its successful development. The load grows from 10 to 75 simultaneous requests, and he does not care for such a load.

Suddenly, one of the nights several thousand letters with the following contents arrive in the mail:

Traceback:
   ...
   > link = LinkModel.objects.get (url = url)
OperationalError: FATAL: remaining connection slots are reserved for non-replication superuser connections

And this is good if you set the en_US.UTF-8 locale to postgresql.conf , because if you used the default Ubuntu / Debian configuration, you will get a thousand emails with a message like:

OperationalError: ?????: ?????????? ????? ???????????? ???????????????? ??? ???????????? ?????????????????? (?? ??? ???????????)

The application created too many connections to the database (by default - a maximum of 100 connections), for which it was punished.

Here is the very first pitfall: Django does not have a database connection pool , because it is simply not needed in synchronous code. A single synchronous Django process cannot process requests in parallel, it serves only one request at any time, and therefore it does not need to create more than one database connection.


There is only one solution: we urgently need a pool of database connections.

There are relatively few pool implementations for Django, for example django-db-pool and django-psycopg2-pool . The first pool is based on psycopg2.TreadedConnectionPool , which throws an exception when trying to take a connection from an empty pool. The application will behave the same as before, but other applications will be able to create a connection to the database. The second pool is based on gevent.Queue : when you try to take a connection from an empty pool, the greenlet will be blocked until another greenlet puts the connection into the pool.
Most likely you will choose the second solution as a more logical one.

Database Queries Inside Greenlets

We have already patched the application using gevent and we have few synchronous calls, so why not squeeze the most out of the greenlets? We can make several HTTP requests in parallel or create subprocesses. We might want to use the database in the greenlets:

def some_view(request):
    greenlets = [gevent.spawn(handler, i) for i in xrange(5)]
    gevent.joinall(greenlets)
    return HttpResponse("Done")
def handler(number):
    obj = MyModel.objects.get(id=number)
    obj.response = send_http_request_somewhere(obj.request)
    obj.save(update_fields=['response'])

Several hours passed, and suddenly our application completely stopped working: for any request we get 504 Gateway Timeout . What happened this time? You'll have to read some Django code for clarification.

All connections are stored in django.db.connections , which is an instance of the django.db.utils.ConnectionHandler class . When the ORM is ready to make a request, it requests a connection to the database by calling connections ['default'] . ConnectionHandler .__ getattr__ in turn checks for a connection in ConnectionHandler._connections , and if it is empty, it creates a new connection.

All open connections must be closed after use. The request_finished signal , which runs in django.http.HttpResponseBase.close, does this . Django closes database connections at the very last moment, when no one will contact them, which is quite logical.

The catch is exactly how ConnectionHandler stores database connections. To do this, he uses threading.local , which after mankipping turns into gevent.local.local . Once declared, this data structure works as if it were unique in every greenlet. Some_view controller started to be processed in one greenlet, and in ConnectionHandler._connectionsalready have a connection to the database. We created several new greenlets, in which ConnectionHandlers._connections turned out to be empty, and more connections from the pool were taken for these greenlets. After our new greenlets disappeared, the contents of their local () disappeared , the connections to the database were irretrievably lost and no one would return them back to the pool. Over time, the pool is completely empty.

When developing on Django + gevent, you always need to remember this nuance and close the database connections at the end of each greenlet by calling django.db.close_connection . You need to call it even when an exception occurs, for which you can use a small decorator-contextmanager.



You need to use this wrapper wisely: close all connections before each switching of greenlets (for example, before urllib2.urlopen ), and also make sure that connections are not closed inside an incomplete transaction or loop through an iterator like Model.objects.all () .

We use Django ORM separately from Django

We can comprehend the same problems if we create an analogue of cron or Celery, which will occasionally make queries to the database. The same thing awaits us if you lift Django using gevent.WSGIServer and simultaneously raise any services with a different protocol that Django ORM will use. The main thing is to return connections to the database pool on time, then the application will work stably and bring you joy.

conclusions

In this post, it was said about elementary rules that you must use the database connection pool and that you need to return the connection back to the pool immediately after use. You would definitely consider this if you only used gevent and psycopg2. But Django ORM operates with such high-level abstractions that the developer does not deal with database connections, and over time, these rules can be forgotten and rediscovered.