Hangfire - Task Scheduler for .NET

Image from hangfire.io

Hangfire is a multi-threaded and scalable task scheduler built on the client-server architecture on the .NET technology stack (primarily the Task Parallel Library and Reflection), with intermediate storage of tasks in the database. Fully functional in the free (LGPL v3) open source version. This article describes how to use Hangfire.

The outline of the article:

• Work principles
• Inside and features of the Hangfire server
• Repeatable and deferred tasks
• Quartz.NET Microview
• Pro (non) stress testing
• Conclusions. Poll.

Work principles

What is the point? As you can see on the KDPV, which I honestly copied from the official documentation, the client process adds a task to the database, the server process periodically polls the database and performs tasks. Important points:

• All that connects the client and server is access to a common database and common assemblies in which task classes are declared.
  
• Scaling the load (increasing the number of servers) - yes!
• Without a database (task storage), Hangfire does not work and cannot work. By default, SQL Server is supported; there are extensions for a number of popular DBMSs . The paid version adds support for Redis.
  
• Anything can act as a host for Hangfire: ASP.NET application, Windows Service, console application, etc. all the way to Azure Worker Role.
  

From the client’s point of view, the work with the task takes place according to the “fire-and-forget” principle, or more precisely - “added to the queue and forgot” - nothing happens on the client other than saving the task to the database. For example, we want to execute the MethodToRun method in a separate process:

BackgroundJob.Enqueue(() => MethodToRun(42, "foo"));

This task will be serialized along with the values ​​of the input parameters and stored in the database:

{
    "Type": "HangClient.BackgroundJobClient_Tests, HangClient, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null",
    "Method": "MethodToRun",
    "ParameterTypes": "(\"System.Int32, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089\",\"System.String, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089\")",
    "Arguments": "(\"42\",\"\\\"foo\\\"\")"
}

This information is enough to call the MethodToRun method in a separate process through Reflection, provided that you access the HangClient assembly in which it is declared. Naturally, it is completely optional to keep the code for background execution in the same assembly with the client, in the general case the dependency scheme is as follows:

The client and server must have access to the common assembly, while access to the built-in web interface (about it below) is optional. If necessary, it is possible to replace the implementation of a task already saved in the database - by replacing the assembly referenced by the server application. This is convenient for scheduled tasks, but, of course, it works if the MethodToRun contract in the old and new assemblies is completely identical. The only limitation on the method is the presence of a public modifier.
Need to create an object and call its method? Hangfire will do it for us:

 BackgroundJob.Enqueue(x => x.Send(13, "Hello!"));

And even receive an instance of EmailSender through a DI container if necessary.

It’s nowhere to deploy a server (for example, in a separate Windows Service):

public partial class Service1 : ServiceBase
{
    private BackgroundJobServer _server;
    public Service1()
    {
        InitializeComponent();
        GlobalConfiguration.Configuration.UseSqlServerStorage("connection_string");
    }
    protected override void OnStart(string() args)
    {
        _server = new BackgroundJobServer();
    }
    protected override void OnStop()
    {
        _server.Dispose();
    }
}

After the service starts, our Hangfire server will start pulling up tasks from the database and performing them.

Optional to use, but useful and very pleasant is the built-in web dashboard, which allows you to control the processing of tasks:

Inside and features of the Hangfire server

First of all, the server contains its own thread pool implemented through the Task Parallel Library. And the basis is the well-known Task.WaitAll (see the BackgroundProcessingServer class ).

Horizontal scaling? Web farm? Web garden? It is supported:

You don't want to consume additional Thread Pool threads with background processing - Hangfire Server uses custom, separate and limited thread pool.
You are using Web Farm or Web Garden and don't want to face with synchronization issues - Hangfire Server is Web Garden / Web Farm friendly by default.

We can create an arbitrary number of Hangfire servers and not think about their synchronization - Hangfire ensures that one task will be performed by one and only one server. An example implementation is the use of sp_getapplock (see the SqlServerDistributedLock class ).
As already noted, the Hangfire server is not demanding on the host process and can be deployed anywhere from the Console App to the Azure Web Site. However, it is not omnipotent, therefore, when hosting in ASP.NET, a number of common IIS features should be taken into account, such as process recycling , auto-start (startMode = “AlwaysRunning”), etc. However, the planner documentation provides comprehensive information for this case as well.
By the way! I cannot but note the quality of the documentation - it is beyond praise and is located somewhere in the ideal region. The source code of Hangfire is open and high-quality, there are no obstacles to lifting the local server and walking around the code with the debugger.

Repeatable and deferred tasks

Hangfire allows you to create repeatable tasks with a minimum interval per minute:

RecurringJob.AddOrUpdate(() => MethodToRun(42, "foo"), Cron.Minutely);

Run the task manually or delete:

RecurringJob.Trigger("task-id");
RecurringJob.RemoveIfExists("task-id");

Postpone the task:

BackgroundJob.Schedule(() => MethodToRun(42, "foo"), TimeSpan.FromDays(7));

Creating a repeating AND deferred task is possible using CRON expressions (support is implemented through the NCrontab project ). For example, the following task will be performed every day at 2:15 am:

RecurringJob.AddOrUpdate("task-id", () => MethodToRun(42, "foo"), "15 2 * * *");

Quartz.NET Microview

A story about a specific task scheduler would be incomplete without mentioning worthy alternatives. On the .NET platform, an alternative is Quartz.NET , the port of the Quartz scheduler from the Java world. Quartz.NET solves similar problems, like Hangfire - it supports an arbitrary number of “clients” (adding a task) and “servers” (performing a task) using a common database. But the execution is different.
My first acquaintance with Quartz.NET could not be called successful - the source code taken from the officially GitHub repository simply did not compile until I manually corrected the links to several missing files and assemblies (disclaimer: I just tell you how it was). There is no separation of client and server parts in the project - Quartz.NET is distributed as a single DLL. In order for a specific application instance to allow you to only add tasks, and not execute them, you need to configure it .
Quartz.NET is completely free, out of the box offers task storageboth in-memory and using many popular DBMSs (SQL Server, Oracle, MySQL, SQLite, etc.). In-memory storage is essentially an ordinary dictionary in the memory of one single server process that performs tasks. It is possible to implement several server processes only when saving tasks to the database. For synchronization, Quartz.NET does not rely on the specific features of the implementation of a particular DBMS (the same Application Lock in SQL Server), but uses one generalized algorithm. For example, by registering in the QRTZ_LOCKS table, one-time operation of no more than one scheduler process with a specific unique id is guaranteed, the task is issued by simply changing the status in the QRTZ_TRIGGERS table.

The task class in Quartz.NET must implement the IJob interface:

public interface IJob
{
    void Execute(IJobExecutionContext context);
}

With this limitation, it is very simple to serialize the task: the full class name is stored in the database, which is enough for the subsequent receipt of the type of the task class via Type.GetType (name). To transfer parameters to the task, the JobDataMap class is used, and parameters of an already saved task can be changed.
As for multithreading, Quartz.NET uses classes from the System.Threading namespace: new Thread () (see the QuartzThread class ), its thread pools, synchronization through Monitor.Wait / Monitor.PulseAll.
A considerable spoon of tar is the quality of official documentation. For example, here is the clustering material: Lesson 11: Advanced (Enterprise) Features. Yes, yes, that's all there is on the official website on this topic. Somewhere in the vast expanses of SO there was an enchanting advice to also look at the guides for the original Quartz , there the topic is disclosed in more detail. The desire of developers to support a similar API in both worlds - Java and .NET - cannot but affect the speed of development. Quartz.NET releases and updates are infrequent.

The main characteristics of the two planners considered are summarized in the table:

Characteristic	Hangfire	Quartz.NET
Unlimited number of clients and servers	Yes	Yes
Source	github.com/HangfireIO	github.com/quartznet/quartznet
Nuget package	Hangfire	Quartz
License	LGPL v3	Apache License 2.0
Where is the host	Web, Windows, Azure	Web, Windows, Azure
Task storage	SQL Server (by default), a number of DBMS through extensions , Redis (in the paid version)	In-memory, a number of databases (SQL Server, MySQL, Oracle ...)
Multithreading implementation	Tpl	Thread, Monitor
Web interface	Yes	Not. Planned in future versions.
Deferred tasks	Yes	Yes
Repeatable tasks	Yes (minimum interval 1 minute)	Yes (minimum interval 1 millisecond)
Cron expressions	Yes	Yes

UPDATE: As ShurikEv rightly pointed out in the comments, a web-interface for Quartz.NET exists: github.com/guryanovev/CrystalQuartz

Pro (non) stress testing

It was necessary to check how Hangfire would cope with a large number of tasks. No sooner said than done, and I wrote a simple client that adds tasks with an interval of 0.2 s. Each task writes a line with debugging information to the database. Having set a 100K task limit on the client, I launched 2 client instances and one server, and the server with the profiler (dotMemory). After 6 hours, 200K of successfully completed tasks in Hangfire and 200K of added rows in the database were already waiting for me. The screenshot shows the profiling results - 2 snapshots of the memory status “before” and “after” the execution:

At the next stages, 20 client processes and 20 server processes worked, and the task execution time was increased and became a random variable. That's just on Hangfire it was not reflected at all in any way:

Conclusions. Poll.

Personally, I liked Hangfire. A free, open source product that reduces the cost of developing and maintaining distributed systems. Do you use anything like that? I invite you to participate in the survey and tell your point of view in the comments.