Deferred: all the details

In a previous article , the basic principles of Deferred and its application in asynchronous programming were described. Today we will try to consider in detail the functioning of Deferred and examples of its use.

So, Deferred is a delayed result, a result of execution that will become known after a while. The result stored in Deferred may be an arbitrary value (successful execution) or an error (exception) that occurred during the execution of an asynchronous operation. Since we are interested in the result of the operation and we received from some asynchronous function Deferred, we want to perform actions at the moment when the result of the execution is known. Therefore, Deferred, in addition to the result, also stores a chain of handlers: result handlers (callback) and error handlers (errback).

Let's consider in more detail a chain of handlers:



Handlers are arranged in “layers” or levels; execution occurs clearly in levels from top to bottom. At the same time, callback and errback handlers are located at each level, one of the elements may be absent. At each level, either callback or errback can be executed, but not both. Execution of handlers occurs only once, there can be no re-entry.

Callback handler functions are functions with one argument - the result of execution:

def callback(result):
    …

The errback handler functions accept as an argument an exception wrapped in the Failure class :

def errback(failure):
    …

Deferred execution begins with the result that appears in Deferred: successful execution or exception. Depending on the result, the corresponding branch of handlers is selected: callback or errback. After that, the next level of handlers is searched in which the corresponding handler exists. In our example in the figure, a successful execution result was obtained and the result was passed to the handler callback1.

Further execution leads to the call of the handlers at the lower levels. If a callback or errback ends with a return value that is not Failure, the execution is considered successful and the result is sent to the input by the callback handler at the next level. If, during the execution of the handler, an exception was thrown or a Failure value was returned, errback will be transferred to the control at the next level, which will receive the exception as a parameter.

In our example, the handler was callback1successful, its result was passed to the handler callback2, in which an exception was thrown, which led to the transition to the chain of errback-handlers, at the third level, the errback handler was absent and the exception was passed toerrback4that handled the exception returned a successful execution result, which is now the result of Deferred, however there are no more handlers. If another level of handlers is added to Deferred, they will be able to access this result.

Like all other Python objects, a Deferred object lives as long as there are references to it from other objects. Usually, the object that returned Deferred saves it, because he needs to transfer the result to Deferred at the end of the asynchronous operation. More often than not, other participants (adding event handlers) do not save references to Deferred, so the Deferred object will be destroyed at the end of the chain of handlers. If a Deferred destruction occurs, in which an unhandled exception remains (execution ended with an exception and there are no more handlers), a debug message is printed with a traceback exception. This situation is similar to “popping” an unhandled exception to the top level in a regular synchronous program.

Deferred Squared

The return value of callback and errback can also be another Deferred, then execution of the chain of handlers of the current Deferred is suspended until the end of the chain of handlers of the nested Deferred.



In the example shown in the figure, the callback2 handler returns not the usual result, but another Deferred - Deferred2. In this case, the execution of the current Deferred is suspended until the result of the execution of Deferred2 is received. The result of Deferred2 - success or exception - becomes the result passed to the next level of the first Deferred handlers. In our example, Deferred2 ended with an exception that will be passed to the input of the errback2first Deferred handler .

Errback exception handling

Each errback exception handler is an analog of the try..except block, and the except block usually responds to some type of exception, this behavior is very easy to reproduce using Failure :

def errback(failure):
    """
    @param failure: ошибка (исключение), завернутое в failure
    @type failure: C{Failure}
    """
    failure.trap(KeyError)
    print "Got key error: %r" % failure.value
    return 0

The trapclass method Failurechecks whether the exception wrapped in it is the heir or the class itself KeyError. If this is not the case, the original exception is thrown again, interrupting the execution of the current errback, which will lead to the transfer of control to the next errback in the handler chain, which imitates the behavior of the block exceptin case of an exception type mismatch (control is transferred to the next block). The property valuestores the original exception, which can be used to obtain additional information about the error.

Note that the errback handler must complete in one of two ways:

1. Return some value that will become the input value of the next callback, which in meaning means that the exception has been processed.
2. Throw an original or new exception - the exception was not processed or a new exception was thrown, the errback chain continues.

There is a third option - return Deferred, then the further execution of the handlers will depend on the result of Deferred.

In our example, we handled the exception and passed 0 as the result (for example, the absence of some key is equivalent to its zero value).

Getting ready for asynchrony in advance

As soon as asynchrony appears, that is, some function instead of the immediate value returns Deferred, asynchrony begins to spread across the tree of functions above, forcing Deferred to return from functions that were previously synchronous (returning the result directly). Consider a conditional example of such a transformation:

def f():
    return 33
def g():
    return f()*2

If for some reason the function fcannot return the result immediately, it will start returning Deferred:

def f():
    return Deferred().callback(33)

But now the function is gforced to return Deferred, catching on a chain of handlers:

def g():
    return f().addCallback(lambda result: result*2)

A similar scheme of “conversion” occurs with real functions: we get Deferred results from the function calls in the tree, attach our callback handlers to their Deferred, which correspond to the old, synchronous code of our function, if we had exception handlers, we add and errback handlers.

In practice, it is better to first identify those parts of the code that are asynchronous and will use Deferred, than to remodel synchronous code into asynchronous. Asynchronous code starts with those calls that cannot build the result directly:

• network input-output;
• access to DBMS network services, memcached;
• remote RPC calls;
• operations whose execution will be highlighted in a thread in the Worker model, etc.

In the process of writing an application, it is often clear that there will be asynchronous access at this point, but it is not there yet (an interface with a DBMS, for example, is not implemented). In this situation, you can use the functions defer.successor defer.failto create a Deferred, which already contains the result. Here's the shortest way to rewrite a function f:

from twisted.internet import defer
def f():
    return defer.success(33)

If we don’t know whether the called function will return the result synchronously or return Deferred, and we don’t want to depend on its behavior, we can wrap the call to it in defer.maybeDeferredwhich any option will make it equivalent to calling Deferred:

from twisted.internet import defer
def g():
    return defer.maybeDeferred(f).addCallback(lambda result: result*2)

This version of the function gwill work with both synchronous and asynchronous f.

Instead of a conclusion

You can talk about Deferred for a very long time, as an additional reading I can again recommend the list of materials at the end of the previous article .