October 4, 2012 at 18:17

The shortest record of asynchronous calls in tornado or patch bytecode in decorator

    The complex asynchronous handler in tornado sometimes spreads to dozens of callback functions, which makes it difficult to perceive and modify code. Therefore, there is a tornado.gen module that allows you to write a handler as a generator. But a lot of yield gen.Task (...) doesn't look very good either. Therefore, in a fit of delirium, I wrote a decorator that simplifies writing:
    BeforeAfter
    @asynchronous
@gen.engine
def get(self):
    result, status = yield gen.Task(
        db.users.find_one, {
            '_id': ObjectId(user_id),
        },
    )

    @asynchronous
@gen.engine
@shortgen
def get(self):
    result, status << db.users.find_one_e({
        '_id': ObjectId(user_id),
        },
    )


    How it works


    As you already noticed, we replaced yield with << . Since python will not allow us to do this with standard tools, we need to modify the bytecode. For simple work with it, we will use the Byteplay module . Let's see the bytecode of two simple functions:
    from byteplay import Code
from pprint import pprint
def gen():
    a = yield 1
pprint(Code.from_code(gen.func_code).code)

    [(SetLineno, 5),  # переходим на 5 строку
 (LOAD_CONST, 1), # загружаем константу 1
 (YIELD_VALUE, None), # "отдаём" загруженное значение
 (STORE_FAST, 'a'), # записываем в переменную a
 (LOAD_CONST, None),
 (RETURN_VALUE, None)]

    def shift():
    a << 1
pprint(Code.from_code(shift.func_code).code)

    [(SetLineno, 10),
 (LOAD_GLOBAL, 'a'),  # a из глобального пространства
 (LOAD_CONST, 1), # загружаем константу 1
 (BINARY_LSHIFT, None), # делаем сдвиг влево для a
 (POP_TOP, None),  # убираем верхний элемент стека
 (LOAD_CONST, None),
 (RETURN_VALUE, None)]

    Therefore, we will make a simple patcher purely for this situation:
    from byteplay import YIELD_VALUE, STORE_FAST
code = Code.from_code(shift.func_code)
code.code[3] = (YIELD_VALUE, None)
code.code[4] = (STORE_FAST, 'a')
code.code.pop(1)
pprint(code.code)

    [(SetLineno, 10), 
 (LOAD_CONST, 1),
 (YIELD_VALUE, None),
 (STORE_FAST, 'a'),
 (LOAD_CONST, None),
 (RETURN_VALUE, None)]

    Now we have a bytecode almost identical to the bytecode of the gen function , apply it to shift and check the result:
    shift.func_code = code.to_code()
res_gen = gen().send(None)
res_shift = shift().send(None)
print res_gen
print res_shift
print res_gen == res_shift

    1
1
True

    The result is the same. The code for the general situation can be viewed on github . You can learn more about bytecode in the official documentation . In the meantime, we will return to tornado. Take the ready-made shortgen decorator . And write a simple handler:

    def fetch(callback):
    callback(1)
class Handler(BaseHandler):
    @asynchronous
    @gen.engine
    @shortgen
    def get(self):
        result << gen.Task(fetch)

    The code has become a little better, but we still have to manually wrap the call in gen.Task , so we ’ll create another decorator to automate this process:

    def fastgen(fnc):
    return partial(gen.Task, fnc)
@fastgen
def fetch(callback):
    callback(1)
class Handler(BaseHandler):
    @asynchronous
    @gen.engine
    @shortgen
    def get(self):
        result << fetch()

    Now everything looks pretty decent, but how will it work with third-party libraries? But nothing, so now we need to patch them! No, we won’t patch the bytecode now, but we’ll just use the monkey patch. In order not to break the old code, we replace __getattribute__ for the necessary classes with:

    def getattribute(self, name):
    attr = None
    if name.find('_e') == len(name) - 2:
        attr = getattr(self, name[:-2])
    if hasattr(attr, '__call__'):
        return fastgen(attr)
    else:
        return super(self.__class__, self).__getattribute__(name)

    Now if the patched object does not have an attribute, for example, find_e (the postfix _e was added so as not to break the old code), we will return the find attribute wrapped in the fasttgen decorator .
    And now the code, for example, for asyncmongo, will look like this:

    from asyncmongo.cursor import Cursor
Cursor.__getattribute__ = getattribute
class Handler(BaseHandler):
    @asynchronous
    @gen.engine
    @shortgen
    def get(self):
        result, status << self.db.posts.find_e({'name': 'post'})

    How to use it


    First, install the resulting module:

    pip install -e git+https://github.com/nvbn/evilshortgen.git#egg=evilshortgen

    Now patch the classes we need:

    from evilshortgen import shortpatch
shortpatch(Cls1, Cls2, Cls3)

    Wrap our own asynchronous methods and functions in the decorator:

    from evilshortgen import fastgen
@fastgen
def fetch(id, callback):
    return callback(id)

    And use the handler:

    from evilshortgen import shortgen
class Handler(BaseHandler):
    @asynchronous
    @gen.engine
    @shortgen
    def get(self, id):
        data << fetch(12)
        num, user << Cls1.fetch()

    Known Issues


    A call can only set values ​​for variables:

    a << fetch()  # работает
self.a << fetch()  # не работает

    Complex decompressions are not supported:

    a, b << fetch()  # работает
(a, b), c << fetch()  # не работает

    References


    Evilshortgen on github
    Details about Byteplay bytecode
    Tags:

    Also popular now: