Celery vs RQ for Batch Task Queuing: Choosing a Broker-Backed Queue for Archival Ingest

Choosing the queue that carries your ingest batches is an architecture decision you make once and live with for a decade, because it shapes what happens the moment a worker dies mid-migration with an irreplaceable master in flight. This decision guide sits under the Async Task Queuing for Batches stage and compares the two Python queues most teams actually shortlist — Celery and RQ — against the criteria that matter for cultural-heritage preservation rather than for a generic web backend. Celery is the heavyweight with strong delivery guarantees and a workflow algebra; RQ (Redis Queue) is the small, legible library you can read end to end in an afternoon. The right answer is not universal: it depends on whether a batch carries reproducible derivatives or the one surviving copy of a document, and this guide gives you the matrix and the verdict to decide.

# What the Decision Actually Turns On

The framing question is not “which queue is faster” but “when a worker is killed with a task half-finished, does the batch come back?” Every other criterion — broker choice, memory footprint, workflow composition, observability — is secondary to that single reliability property, because a lost throughput point costs you a re-scan you can schedule, while a lost master costs you an object that no longer exists. Six criteria separate the two tools in practice:

1. Redelivery on worker death. Celery, configured with acks_late=True and task_reject_on_worker_lost=True, holds the broker message unacknowledged until the task body returns, so a kill -9 mid-task re-queues the work. This is the same late-acknowledgment discipline detailed in Implementing Celery for Asynchronous Ingestion Tasks. RQ’s default worker acknowledges by popping the job before it runs; a hard-killed worker leaves the job neither in the queue nor in the failure registry unless you run it under rq worker with maintenance/--with-scheduler semantics or a monitoring horse — its at-least-once guarantee is weaker out of the box.
2. Broker substrate. Celery speaks AMQP (RabbitMQ) or Redis (and others); RabbitMQ gives you consumer acknowledgments, publisher confirms, and durable queues as first-class broker features. RQ is Redis-only by design, which is simpler to operate but concentrates every durability question on how you have configured Redis persistence (AOF vs RDB).
3. Throughput and memory on large masters. Both can pin one job per worker. Celery’s prefork pool with worker_prefetch_multiplier=1 and worker_max_tasks_per_child recycles workers to release memory leaked by imaging libraries; RQ forks a fresh process per job by default, which gives clean memory reclamation for free but pays a fork cost per TIFF.
4. Workflow composition. Celery canvas — chains, groups, and chords — expresses “validate, then fan out capture and metadata in parallel, then join” natively. RQ expresses dependencies with Queue.enqueue(..., depends_on=job), a simpler DAG that covers linear and fan-in shapes but has no built-in chord/callback-after-group primitive.
5. Operational complexity. RQ has far fewer moving parts, one dashboard, and a codebase a single engineer can audit. Celery has more configuration surface — and more places to misconfigure a reliability guarantee.
6. Observability. Celery has Flower, events, and mature APM integrations; RQ has rq info, RQ Dashboard, and the failure registry as a legible source of truth.

# Comparison Matrix

The table condenses those criteria into a side-by-side verdict weighted for preservation workloads, where an irreplaceable master must never be acknowledged before the work that protects it has completed.

# Decision Path: Which Queue for Which Requirement

The decision tree below encodes the verdict as a set of gates. The first gate is the only one that is non-negotiable for preservation: if the batch carries irreplaceable masters, broker-enforced redelivery is mandatory and the path leads to Celery regardless of how much you would prefer RQ’s simplicity.

The first gate decides most preservation cases: irreplaceable masters demand broker-enforced redelivery, which points to Celery even when RQ would otherwise be simpler.

# Celery: Late-Acknowledged Task

The Celery side of the comparison is a task that cannot be lost to a worker crash. acks_late=True defers the broker acknowledgment until the body returns, and reject_on_worker_lost=True re-queues the message if the worker process is killed outright rather than letting the dying process ack it. The task is idempotent, so a redelivery re-records the identical SHA-256 outcome instead of forking the provenance record — the same fixity discipline the pipeline applies before promotion to archival storage.

import hashlib
import logging

from celery import Celery

logger = logging.getLogger("archival.queue.celery")

app = Celery("archival_ingest")


def compute_sha256(file_path: str) -> str:
    """Stream a preservation master through SHA-256 without loading it whole."""
    digest = hashlib.sha256()
    with open(file_path, "rb") as handle:
        for chunk in iter(lambda: handle.read(1_048_576), b""):
            digest.update(chunk)
    return digest.hexdigest()


@app.task(
    bind=True,
    acks_late=True,
    reject_on_worker_lost=True,
    max_retries=3,
    autoretry_for=(ConnectionError, TimeoutError, OSError),
    retry_backoff=30,
)
def verify_master(self, file_path: str, expected_hash: str) -> dict:
    """Idempotent fixity check; redelivery after a crash re-records the same result."""
    actual_hash = compute_sha256(file_path)
    verified = actual_hash == expected_hash
    logger.info(
        "celery_fixity_checked",
        extra={
            "task_id": self.request.id,
            "file": file_path,
            "expected": expected_hash,
            "actual": actual_hash,
            "verified": verified,
        },
    )
    if not verified:
        raise ValueError(f"Checksum mismatch for {file_path}: {actual_hash} != {expected_hash}")
    return {"status": "verified", "file": file_path, "hash": actual_hash}

# RQ: Enqueue and Failure-Registry Inspection

The RQ side does the same fixity work but leans on RQ’s most valuable operational feature — the FailedJobRegistry. RQ does not hide failures; a job that raises lands in an enumerable registry you can inspect, triage, and requeue. This is the escalation surface RQ offers instead of Celery’s broker-level redelivery, and it dovetails with the quarantine patterns in Error Handling & Retry Logic. Enqueue carries a Retry policy with explicit backoff intervals, and the inspection routine walks the registry to surface every terminally failed master for an operator.

import logging

from redis import Redis
from rq import Queue, Retry
from rq.job import Job
from rq.registry import FailedJobRegistry

logger = logging.getLogger("archival.queue.rq")

redis_conn = Redis(host="redis", port=6379, db=0)
ingest_queue = Queue("ingest", connection=redis_conn)


def enqueue_master(file_path: str, expected_hash: str) -> str:
    """Enqueue a fixity check with bounded retries and structured logging."""
    job: Job = ingest_queue.enqueue(
        "archival.tasks.verify_master_rq",
        file_path,
        expected_hash,
        retry=Retry(max=3, interval=[30, 60, 120]),
        job_timeout=86_400,  # must exceed the longest master's checksum/migration time
    )
    logger.info(
        "rq_master_enqueued",
        extra={"job_id": job.id, "file": file_path, "expected": expected_hash},
    )
    return job.id


def inspect_failures() -> list[dict]:
    """Enumerate terminally failed jobs for triage and possible requeue."""
    registry = FailedJobRegistry(queue=ingest_queue)
    failures: list[dict] = []
    for job_id in registry.get_job_ids():
        job = Job.fetch(job_id, connection=redis_conn)
        record = {
            "job_id": job.id,
            "func": job.func_name,
            "args": job.args,
            "exc": (job.exc_info or "").strip().splitlines()[-1] if job.exc_info else None,
        }
        failures.append(record)
        logger.warning("rq_job_failed", extra=record)
    return failures

# Validation and Verification

A matrix on paper proves nothing; confirm the reliability property that drove the decision by exercising a real crash on whichever queue you shortlist:

• Kill a worker mid-task and check for the batch. Under Celery with acks_late=True and reject_on_worker_lost=True, kill -9 on a worker running verify_master must leave the message unacked and see it complete on another worker. Under default RQ, the same kill can drop the job — confirm whether your configuration recovers it or requires an external monitor.
• Prove idempotency. Run the fixity task twice on the same master and assert both invocations record the identical SHA-256, so a redelivery cannot corrupt provenance.
• Walk the failure surface. For RQ, force a checksum mismatch and confirm the job appears in inspect_failures(); for Celery, confirm the failure is captured as a durable preservation event, not just a log line, so it folds into object provenance. Both must be validated against the manifest contract described in Batch Validation Schemas, which rejects malformed payloads before they ever reach a worker.
• Measure memory under load. Enqueue a run of worst-case bound-volume TIFF masters and watch resident set size; a leak that RQ’s fork-per-job reclaims for free may require worker_max_tasks_per_child on Celery.

# Edge Cases and Gotchas

Three archival-specific traps apply to both tools. First, neither queue’s result store is an audit system of record — Celery result backends and RQ registries both expire, so preservation outcomes must be persisted to durable storage and reconciled into provenance. Second, a poison message — a corrupt manifest or an unreadable master — will exhaust retries on either queue and hammer the storage array; route terminally failed work to a quarantine rather than retrying forever. Third, Redis-only deployments concentrate risk: if RQ is your queue, Redis durability configuration is not optional, because a Redis flush takes your queue, your results, and your failure registry with it.

# Frequently Asked Questions

# Is RQ ever the right choice for a preservation repository?

Yes, when the batches carry reproducible derivatives rather than irreplaceable masters — thumbnail generation, DIP assembly, re-runnable OCR — and the pipeline is linear or a simple fan-in. RQ’s small codebase, single Redis dependency, and legible FailedJobRegistry are genuine operational advantages for a small team, and losing a derivative job costs a cheap re-run. The line to hold is the first decision gate: the moment a batch carries the one surviving copy of an object, broker-enforced redelivery becomes mandatory and the choice shifts to Celery.

# Can RQ match Celery’s crash-safety?

Partially, and only with effort. RQ’s default worker can drop a hard-killed job, so you must run it with the maintenance and monitoring that detect and requeue orphaned work, and you must configure Redis persistence so the queue survives a broker restart. Even then you are reconstructing, at the application layer, a redelivery guarantee that Celery gets from the broker with acks_late and reject_on_worker_lost. For irreplaceable masters, buying that guarantee at the broker level is more defensible under an ISO 16363 audit than bolting it on above the queue.

# How do the two compare on memory for large TIFF masters?

RQ forks a fresh process per job, so memory leaked by an imaging library is reclaimed automatically when the fork exits — clean, at the cost of a fork per master. Celery’s prefork workers are long-lived, so you bound memory with worker_prefetch_multiplier=1 to reserve one master at a time and worker_max_tasks_per_child to recycle a worker before leaks accumulate. Both can be made safe on multi-gigabyte masters; RQ gets there by default, Celery by configuration.

# Which queue handles a fan-in workflow better?

Celery, unambiguously. Its canvas expresses “validate, then run capture and metadata extraction in parallel, then fire an enrichment callback only after both finish” as a chord over a group, with the join semantics built in. RQ models dependencies with depends_on, which handles linear chains and simple fan-in but has no native chord primitive, so a callback-after-group has to be emulated. If your ingest DAG is genuinely parallel with join points, Celery’s workflow algebra removes a class of orchestration bugs.

# Related

• Async Task Queuing for Batches — the parent stage: queue contract, broker selection, and the batch manifest schema both tools consume.
• Implementing Celery for Asynchronous Ingestion Tasks — the full Celery reliability build: late acknowledgment, broker tuning, and canvas workflows.
• Error Handling & Retry Logic — dead-letter routing, backoff policy, and quarantine patterns for terminally failed batches on either queue.