A simple task server with a queue in MySQL (no lock issues)

    In almost every more or less dynamic project, there is a need to perform task queues in the background (sending email, updating the cache, reindexing the search, etc.). Job servers (Gearman, etc.) are good, but for most simple tasks they are redundant. The classic implementation of queues in MySQL (using SELECT ... LOCK FOR UPDATE) with increasing load over time begins to lead to problems with locking. Therefore, as it usually happens, I had to write my own “bicycle” for working with background tasks, which would “work exactly” and be extremely simple.

    Basis: Cron, PHP 5.3 (mysqli), MySQL> 5.1 - it is easy to stick to almost any hosting.
    The operation of obtaining (capturing) a task is atomic (one UPDATE request). No problems with locking and RC.
    Possibility of distributing tasks to groups of workers and priorities, transferring an array of data into an executable method (function).
    Three modes of processing completed tasks: move the record to a separate table, delete the record, leave the record and mark it as successfully processed.
    Processing incomplete tasks or tasks processed with an error is on the conscience of the developer.
    For everything about everything 400 lines of code (with full PHPDOC).
    Limitations: the current implementation is not suitable for persistent connections, but if someone needs it, it's easy to finish it. Even if you want to rewrite to another language :)

    The possibility of non-blocking work with the queue is realized through the use of user variables in the UPDATE query with their subsequent selection. Devoting an entire article to this technique is silly. The final implementation that can be applied to the case is much more pleasant (We are practicing with you, aren't we?). In all other respects, it is an exclusively classic lineup with groups and priorities.

    Usage example (client):
    $task_server = \DBTaskServer::create('localhost', 'root', '', 'testDB', 'jobs_queue');
    $task_server->addTask('mywork', $data);
    

    mywork is a function that should be available to the worker. The $ data array will be passed to it . It is also possible to specify a call to static class methods.
    $task_server->addTask('MyWork::doWork', $data);
    


    Worker example:
    \DBTaskServer::create('localhost', 'root', '', 'testDB', 'jobs_queue') // Создаем сервер.
    		->setByCLIAgruments($argv) // Устанавливаем параметры вызова из консоли.
    		->setMode(\DBTaskServer::MODE_MARK_AS_COMPLETED) // Выбираем режим обработки.
    		->run(); // Запускам воркера.
    


    Launching a worker from the console with the parameters:
    /path/to/script/worker.php [max_tasks_per_lifecycle] [comma_separated_group_ids]
    

    As the name implies, the first option tells you how many tasks the worker can complete before completing the work (unless of course those are available to him), the second option is the group_id values ​​of the tasks that this worker should process. If no groups are specified, the worker processes any groups.

    For instance:
    /path/to/script/worker.php 100 3,5,6
    

    Complete 100 tasks from groups 3, 5 and 6.
    If no tasks are found, the worker will immediately finish his work.

    Add the worker to the crowns:
    0-59/5 * * * * /path/to/script/worker.php 5 3 >/dev/null 2>&1
    

    Every 5 minutes, process 5 jobs with group_id = 3.

    The archive contains examples of a client, a worker, the server class itself (documented), sql file with a task table.
    Download here (as many as 5kB).

    Have a nice code.

    Also popular now: