The uWSGI Spooler



    When you design scalable systems where you have to access many external components, for example, using a third-party API, sending mail or converting video, the best way to implement this is an asynchronous model with a queuing system, which is the link for the interaction of all system components.


    The most popular queuing system in Python is Celery, which has a wide range of task management capabilities. Unfortunately, Celery-based systems are hard to maintain, and when something goes wrong, finding a problem can be quite difficult. You can ask any devop about experience with Celery, but be prepared to hear not very nice words.


    Fortunately, there is an alternative solution - uWSGI Spooler, and in this article I will talk more about it.



    The main difference from Celery is that you do not need to use additional components (Celery itself and storage, for example Redis), so the number of failure points is reduced by two. A directory, an external directory, or a network pool can be used as a task store.


    To control Python programs, we often use uWSGI. Why? Because it is easy to configure, reliable, flexible and fulfills most of the requirements.


    In addition to serving Python code in the form of providing continuous access to a web application, uWSGI also includes the Spooler component , which implements a queue system. Spooler has some features, and its documentation is rather scarce.


    Using uWSGI Spooler is easy, one-two-three! But there are a few nuances.
    The uwsgi module cannot be imported from the code, and accordingly the code cannot be tested from the console, you must run the uwsgi worker every time, for which you need to create a config:


    [uwsgi]
    socket = /var/run/mysite.sock 
    master = True
    processes = 4
    project_dir = /home/myuser/mysite
    chdir = %(project_dir)
    spooler = /var/uwsgi_spools/mysite_spool
    spooler-import = path.to.spool.package # (package to import spool file)
    spooler-frequency = 10 # Frequency for scanning spool
    max-requests = 5000
    module = wsgi:application
    touch-reload = wsgi.py

    Worker File:


    from uwsgidecorators import spool, uwsgi
    @spool
    def my_func(args):
        print(args)
        # do some job

    Statement of the problem from your code:


    import uwsgi_spools.mysite_spool as mysite_spool
    mysite_spool.my_func.spool(test=True)

    As you can see from the example, the entry threshold for use is very low.


    Inside the task, one argument is available that contains a dictionary with three service keys (function name ud_spool_func , task name spooler_task_name, task status ud_spool_ret ) and all the parameters that were passed when creating the task, in the example it is the test key .


    Task can return three statuses:


    • -2 (SPOOL_OK) - the task is completed, it will be removed from the queue;
    • -1 (SPOOL_RETRY) - something went wrong, the task will be called again;
    • 0 (SPOOL_IGNORE) - ignore task.

    All other values ​​will be interpreted as -1 (SPOOL_RETRY).


    Feature: the decorator @spoolis executed once (returns SPOOL_OK) if the function does not fall with an exception.
    In order to manage the life cycle you need to use @spoolraw.


    Special keys (auxiliary) when creating a task:


    • spooler - the absolute path to the spooler that will perform the task;
    • at - unix time, when the task should be completed (more correctly, it will not be completed earlier than this value);
    • priority - indicates a subfolder in the task folder (a larger number of workers can be allocated to such a subfolder), through --spooler-ordered you can configure priorities;
    • body - this key is used for values ​​greater than 64 KB, the task will be available in serialized form.

    In addition to the decorator, a decorator @spoolis also available @timerthat takes the number of seconds as an argument and allows you to execute the decorated function with the specified interval.


    @timer(30)
    def my_func(args):
        print(args)
        # do some job every 30 sec

    Similarly, @timerthere is a decorator @spoolforeverthat will restart the execution of the function (completion of the task with the status SPOOL_RETRY).


    @spoolforever
    def my_func(args):
        print(args)
        # do some job and repeat

    To configure workers for working on the network, you need to add the address at which it will be available in the ini-file:


    socket = 127.0.0.1:10001

    When creating a task, specify the address of the task recipient:


    uwsgi.send_message(“127.0.0.1:10001”, 17, 0, test=True, 5)
    # или
    uwsgi.spool(test=True, spooler=“127.0.0.1:10001”)

    Thus, uWSGI Spooler can be used as a replacement for queues, but if you still lack opportunities or want a little sugar, you can use uwsgi-tasks , which implements the missing.


    Also popular now: