Exponential Backoff With Jitter for Scanner Timeouts

A transient timeout from a scanner controller or a storage array is not the dangerous event — the dangerous event is what fifty ingest workers do about it at the same instant. This walkthrough sits under Error Handling & Retry Logic in the broader Automated Ingestion & Batch Scanning Workflows pipeline, and it isolates one specific failure: a shared device returns a burst of 504/ETIMEDOUT responses, every worker starts a textbook exponential backoff from the same clock, and because the delays are identical the whole fleet retries in lockstep — 1 s later, 2 s later, 4 s later — hammering the already-saturated controller in synchronized waves until it never recovers. The fix is not a longer backoff. It is deliberately randomized backoff, and choosing the right jitter strategy, retry budget, and retryable/terminal classification is what separates a fleet that self-heals from one that DDoSes its own hardware.

# Root-Cause Analysis of the Retry Storm

The thundering herd against a single digitization controller is almost always assembled from three independent mistakes, each of which is individually survivable but collectively catastrophic:

1. Fixed or no-jitter backoff (the synchronizer). A pure base * 2^attempt schedule is deterministic. When a shared resource fails, it fails for all callers within a narrow window, so all callers compute the same delay sequence and re-converge on the device at exactly the same moments. Backoff spreads retries out in time for a single worker but does nothing to spread them out across workers — the peaks simply move together. Two hundred workers backing off in unison still deliver two hundred simultaneous requests; the exponential curve just decides when.
2. Unbounded retries (the amplifier). A retry loop with no attempt ceiling and no total time budget converts a brief controller hiccup into an indefinite siege. Each failed wave adds load that lengthens the outage, which fails more requests, which schedules more waves. Without a cap on both attempt count and wall-clock budget, the retry mechanism becomes a positive-feedback loop that guarantees the resource cannot drain its backlog.
3. Retrying non-transient errors (the poison). Backoff only helps when the underlying condition is temporary. Retrying a 401 Unauthorized, a 400 malformed capture command, or a 507 Insufficient Storage wastes the entire budget on a condition that will never clear, and worse, it presents a healthy device with a storm of requests it is correctly rejecting. Every retry must first answer “is this error even retryable?” before it asks “how long should I wait?”

The core insight, formalized in AWS’s well-known analysis of the problem, is that adding randomness decorrelates the retry times. The base delay still grows exponentially to relieve a struggling device, but each worker draws its actual sleep from a random interval, so the fleet’s retries smear across the time axis instead of stacking into spikes. For full jitter the sleep before attempt $n$ is:

$$ \text{sleep}_n = \operatorname{random}!\left(0,; \min!\left(\text{cap},; \text{base} \cdot 2^{,n}\right)\right) $$

The min(cap, ...) term bounds the exponential so a late attempt cannot schedule an absurd multi-hour wait, and the random(0, ...) term is what actually breaks the synchronization. The diagram below contrasts the two regimes on the same time axis.

Same number of retries, same exponential envelope: without jitter they stack into device-killing spikes; with full jitter they spread across each window and stay under the controller’s safe ceiling.

# Choosing a Jitter Strategy

Three jitter strategies are worth knowing, and they trade off between how aggressively they de-synchronize the fleet and how much they preserve the exponential shape. The route_capture_command layer described in Scanner API Integration & Routing is exactly the code path where these delays are applied before a capture command is re-sent.

Full jitter is the right default for transient scanner and storage timeouts: simulations in the AWS study showed it produces both the fewest total calls and the lowest completion-time variance under contention. Equal jitter is a reasonable choice when a task must not retry too eagerly — it guarantees at least half the exponential delay elapses before the next attempt. Decorrelated jitter, where each delay is derived from the previous one rather than the attempt number, gives the widest spread over long chains and is what pairs naturally with the idempotency guarantees covered in designing idempotent retries for partial batch failures, since a redelivered task may resume mid-chain.

# Step-by-Step Resolution: A Bounded Full-Jitter Retry Utility

The utility below implements full-jitter backoff with three non-negotiable guardrails: a retryable/terminal classifier so the budget is never spent on a permanent error, a bounded budget on both attempt count and wall-clock time, and structured per-attempt logging of the computed delay so a retry storm is visible in the logs before it is visible on the hardware. It also honors a server-supplied Retry-After when present, which overrides the computed jitter.

import logging
import random
import time
from dataclasses import dataclass
from typing import Callable, Optional, TypeVar

logger = logging.getLogger("archival.ingest.retry")

T = TypeVar("T")


class TerminalScannerError(Exception):
    """A non-transient failure (auth, malformed command, out of storage). Never retried."""


class TransientScannerError(Exception):
    """A retryable timeout/5xx from the scanner controller or storage array."""

    def __init__(self, message: str, retry_after: Optional[float] = None) -> None:
        super().__init__(message)
        self.retry_after = retry_after


# HTTP-ish status codes we treat as transient for a shared digitization device.
RETRYABLE_STATUS = frozenset({408, 425, 429, 500, 502, 503, 504})
TERMINAL_STATUS = frozenset({400, 401, 403, 404, 405, 501, 507})


@dataclass(frozen=True)
class RetryPolicy:
    """Bounded full-jitter backoff. Caps BOTH attempts and total wall-clock budget."""

    base: float = 0.5          # seconds — first-attempt backoff ceiling
    cap: float = 30.0          # seconds — max single backoff, bounds the exponential
    max_attempts: int = 6      # hard ceiling on tries (1 initial + 5 retries)
    max_elapsed: float = 120.0 # seconds — total budget across all attempts

    def full_jitter_delay(self, attempt: int) -> float:
        """sleep = random(0, min(cap, base * 2**attempt)) — the herd-breaker."""
        ceiling = min(self.cap, self.base * (2 ** attempt))
        return random.uniform(0.0, ceiling)


def call_with_backoff(
    operation: Callable[[], T],
    policy: RetryPolicy,
    *,
    task_id: str,
) -> T:
    """Invoke a scanner/storage operation with bounded full-jitter retries.

    Retries only TransientScannerError; TerminalScannerError propagates immediately.
    A server-supplied retry_after overrides the computed jitter. Every attempt logs
    its computed delay so a forming retry storm is observable in structured logs.
    """
    started = time.monotonic()
    last_exc: Optional[BaseException] = None

    for attempt in range(policy.max_attempts):
        try:
            result = operation()
            if attempt:
                logger.info(
                    "retry_succeeded",
                    extra={"task_id": task_id, "attempt": attempt,
                           "elapsed_s": round(time.monotonic() - started, 3)},
                )
            return result
        except TerminalScannerError:
            logger.error(
                "retry_abandoned_terminal",
                extra={"task_id": task_id, "attempt": attempt},
            )
            raise
        except TransientScannerError as exc:
            last_exc = exc
            # Respect an explicit Retry-After; otherwise draw full jitter.
            computed = policy.full_jitter_delay(attempt)
            delay = exc.retry_after if exc.retry_after is not None else computed

            elapsed = time.monotonic() - started
            is_last = attempt == policy.max_attempts - 1
            budget_blown = elapsed + delay > policy.max_elapsed

            if is_last or budget_blown:
                logger.error(
                    "retry_budget_exhausted",
                    extra={"task_id": task_id, "attempt": attempt,
                           "elapsed_s": round(elapsed, 3),
                           "reason": "max_attempts" if is_last else "max_elapsed"},
                )
                raise

            logger.warning(
                "retry_scheduled",
                extra={"task_id": task_id, "attempt": attempt,
                       "computed_jitter_s": round(computed, 3),
                       "sleep_s": round(delay, 3),
                       "honored_retry_after": exc.retry_after is not None,
                       "elapsed_s": round(elapsed, 3)},
            )
            time.sleep(delay)

    assert last_exc is not None  # unreachable: loop returns or raises
    raise last_exc

The classifier is the first line of defense: mapping a device response to TerminalScannerError short-circuits the loop so the budget is never wasted on a 401 or 507. Wiring the status sets into whatever HTTP or SANE client the controller speaks is a one-liner — raise TransientScannerError for RETRYABLE_STATUS, TerminalScannerError for TERMINAL_STATUS, and default unknown codes to terminal so a novel error fails loudly rather than storming silently.

def classify_response(status: int, retry_after_header: Optional[str]) -> None:
    """Translate a controller status code into a retryable or terminal exception."""
    if status in TERMINAL_STATUS:
        raise TerminalScannerError(f"non-transient controller status {status}")
    if status in RETRYABLE_STATUS:
        retry_after = float(retry_after_header) if retry_after_header else None
        raise TransientScannerError(f"transient controller status {status}", retry_after)
    if status >= 400:
        # Unknown 4xx/5xx: fail closed rather than risk an unbounded storm.
        raise TerminalScannerError(f"unclassified controller status {status}")

# Validation and Verification

Configuration that looks de-synchronized still needs proof under load. Confirm the fix with a concurrency test and log analysis, not by reading the policy object:

• Assert the delay stays in range. For any attempt, full_jitter_delay(attempt) must fall in [0, min(cap, base*2**attempt)]. Run it a few thousand times per attempt index and assert both bounds — a jitter that can exceed the cap or go negative is a bug that will surface only under production load.
• Measure spread, not just mean. Run 200 concurrent workers against a stub controller that fails the first three calls, then histogram the retry timestamps. Under full jitter no 100 ms bucket should hold more than a small fraction of the fleet; under no-jitter you will see three sharp spikes. The absence of spikes is the verification.
• Prove the budget is bounded. Force perpetual TransientScannerError and assert the call raises retry_budget_exhausted within max_elapsed seconds and after at most max_attempts tries. A retry utility that can loop forever has failed this test regardless of its jitter quality.
• Confirm terminal errors are not retried. Raise TerminalScannerError on the first call and assert exactly one attempt was made and the log line is retry_abandoned_terminal. This guards against the most expensive mistake — spending the whole budget on a permanent condition.

Every retried operation must remain idempotent, so a redelivered capture or checksum records the same outcome; that discipline, established alongside async task queuing for batches, is what makes at-least-once retry safe rather than corrupting.

# Edge Cases and Gotchas

• Retry-After beats your math. A 429 or 503 from a storage array often carries a Retry-After header naming the exact moment it will be ready. Honor it — the server knows more than your exponential curve does — but still clamp it to a sane maximum so a hostile or misconfigured header cannot park a worker for an hour.
• Jitter on a single-worker cron is wasted, and can hurt. Full jitter exists to de-synchronize a fleet. A lone nightly job retrying a mount point gains nothing from randomness and may want equal jitter instead, so it keeps a predictable minimum gap between attempts rather than occasionally retrying almost instantly.
• The cap must exceed one device recovery cycle. If a scanner controller reboots in 45 s but your cap is 30 s, every worker’s maximum backoff still lands mid-reboot and the storm simply resumes. Size cap from the worst-case recovery time of the slowest shared resource, not the average request latency.
• Retry budget must be shorter than the broker visibility timeout. If max_elapsed outlives the queue’s visibility timeout, the broker redelivers the task to a second worker while the first is still backing off — now you have two workers retrying the same object against the same device. Keep the total retry budget comfortably inside the visibility window.
• Clock choice matters. Measure elapsed budget with time.monotonic(), never time.time(). A wall-clock adjustment (NTP step, DST) during a long backoff chain can make a time.time() budget expire early or never — monotonic is immune.

# Frequently Asked Questions

# Why does adding randomness make retries more reliable?

Because the reliability problem is contention, not timing precision. When a shared scanner controller fails, it fails for every worker at once, and a deterministic backoff makes them all recompute the identical delay and re-hit the device in synchronized waves. Randomizing each worker’s delay decorrelates those retry times so the same total number of requests arrives spread across the window instead of stacked into spikes the device cannot absorb. The exponential envelope still backs the fleet off overall; jitter is what stops the peaks from lining up.

# Full jitter, equal jitter, or decorrelated jitter?

Full jitter — random(0, min(cap, base*2**n)) — is the default: it produces the widest spread and, in published simulations, the fewest total calls under contention. Choose equal jitter when a guaranteed minimum gap between attempts matters, since it never retries in near-zero time. Choose decorrelated jitter for long, deep retry chains where deriving each delay from the previous one keeps the spread wide across many attempts. All three beat no-jitter against a shared resource.

# How do I keep retries from becoming an unbounded storm?

Bound two dimensions at once: a maximum attempt count and a maximum wall-clock budget, and stop at whichever comes first. Then classify errors before retrying — permanent failures like 401, 400, or 507 must raise a terminal exception that skips the loop entirely, so the budget is never spent on a condition that will never clear. A retry utility without both a bound and a classifier will, under a real outage, amplify it.

# Should I respect a Retry-After header from the storage array?

Yes, when present it should override your computed jitter, because the server is telling you precisely when it expects to be ready and that beats any client-side guess. Clamp it to a maximum, though, so a misconfigured or malicious header cannot pin a worker indefinitely, and fall back to full jitter whenever the header is absent or unparseable.

# Related

• Error Handling & Retry Logic — the parent area: dead-letter routing, quarantine, and the overall retry policy this backoff plugs into.
• Designing Idempotent Retries for Partial Batch Failures — why every retried operation must be idempotent so a redelivery cannot corrupt a preservation record.
• Scanner API Integration & Routing — the capture-command path where these backoff delays are applied before a request is re-sent to the controller.