Async-background: background tasks in Ruby without Redis and Postgres

Async-background: A Lightweight Background Job System for Ruby with No External Dependencies

Ruby app developers often need to run background jobs, but ready-made solutions like Sidekiq require additional infrastructure. The new gem async-background offers an alternative that runs on Async without Redis or Postgres, with fine-grained resource control.

Why Existing Solutions Fell Short

Sidekiq is typically used for background jobs in Ruby, but it requires Redis. For small projects with just a handful of jobs, that's overkill: an extra container in docker-compose, another point of failure, and infrastructure costs. What's more, Sidekiq doesn't allow flexible load distribution among workers—a heavy nightly report can slow down user email processing.

Other options have their drawbacks too:

• whenever — generates a system crontab, with no queue or app integration.
• rufus-scheduler — runs in separate threads, causing context switching and duplication issues in Async apps with multiple processes.
• good_job and solid_queue — tied to ActiveRecord and Postgres. This creates risks: background jobs compete with transactional load for database resources, potentially slowing the main app.

Thus, there was a need for a solution that combines cron schedules, intervals, and a dynamic queue without external dependencies, plus control over resource utilization.

Key Design Principles

The gem's author outlined six core principles:

• Single event loop. The scheduler runs in the same reactive loop as the app, without separate threads.
• Zero infrastructure. The queue is stored on disk (SQLite in WAL mode or a file in tmp/), no Redis or Postgres.
• Optional dependencies. Extra features (like metrics) are added via optional gems.
• Multi-process safety without a coordinator. Ensures a job runs in only one process when using Falcon (forks).
• Persistence as insurance, speed as a separate mechanism. Reliability comes from SQLite storage, speed from UNIX socket wakeups.
• Managed worker utilization. Developers can assign jobs to specific workers via environment variables and YAML config.

These principles made it possible to build a gem that integrates seamlessly into existing Async apps without extra infrastructure.

Architectural Features

MinHeap Instead of Polling Ticks

Instead of a loop with sleep 1 (waking every second to check all jobs), a binary min-heap (MinHeap) is used. This enables:

• Fetching the next job in O(1) time with peek.
• Updating the heap after running a job in O(log n) time with replace_top.
• Sleeping precisely until the next event, saving CPU resources.

The implementation is just 74 lines long and has no external dependencies.

Two Time Scales

For interval-based jobs (e.g., every: 60), monotonic clocks (CLOCK_MONOTONIC) are used to avoid issues with time sync (NTP). For cron jobs (e.g., cron: "0 3 *"), wall-clock time (Time.now) is used since the schedule is tied to calendar time.

In code, it looks like this:

next_run_at = if task_config[:interval]
  now + jitter + task_config[:interval]
else
  now_wall = Time.now
  wall_wait = task_config[:cron].next_time(now_wall).to_f - now_wall.to_f
  now + jitter + [wall_wait, MIN_SLEEP_TIME].max
end

Sharding via CRC32

When using Falcon (which forks), jobs must run in only one process. Deterministic sharding achieves this: Zlib.crc32(name) % total_workers + 1 is computed for the job name. If it matches the current worker's index, the job is loaded.

assigned = config['worker']&.to_i || ((Zlib.crc32(name) % total_workers) + 1)
next unless assigned == worker_index

This eliminates the need for distributed locks.

Skip-on-Overlap

If a job runs longer than its interval (e.g., 90 seconds for a 60-second interval), the new instance is skipped. Instead, the current job is logged, skipped, and rescheduled for the next slot:

if entry.running
  logger.warn('Async::Background') { "#{entry.name}: skipped, previous run still active" }
  metrics.job_skipped(entry)
  entry.reschedule(monotonic_now)
  heap.replace_top(entry)
  next
end

Queue: SQLite + UNIX Socket

The dynamic queue uses two mechanisms:

• SQLite — source of truth. Jobs are persisted to disk for reliability.
• UNIX socket — for instant notifications. After adding a job to the DB, a signal is sent via the socket so the worker processes it immediately.

Polling every 5 seconds acts as a backup in case of signal loss.

Failure Recovery

If a worker crashes mid-job, the status stays "running." On restart, Queue::Store#recover finds such records and returns them to the queue.

Key Takeaways

• No external service dependencies. Async-background relies only on the filesystem, simplifying deployment.
• Resource control. Isolating workers for different job types (cron, queue) prevents resource contention.
• Reliability. Jobs aren't lost thanks to SQLite persistence and recovery mechanisms.
• Flexibility. YAML and environment variable config make job distribution easy to tune.
• Performance. Min-heap and UNIX sockets deliver low latency and efficient CPU use.

— Editorial Team