Recovering from Scanner Firmware Disconnects Mid-Batch: Checkpointing, Journals, and Idempotent Resume

A production capture run of a 900-page bound volume does not fail politely. Two-thirds of the way through, the scanner’s firmware watchdog fires, the USB bus re-enumerates, or the TCP session to a networked device silently drops — and the coordinator is left holding an ambiguous truth: some pages are safely on disk, some are half-written, and the device’s own on-board page counter no longer agrees with anything the host recorded. This page is the mid-batch recovery reference for the device-control layer specified in Scanner API Integration & Routing: it takes the scan-job contract and dispatch model defined there and answers the one question a naive control loop cannot — where do I safely resume? The failure this page solves is the gap between “the batch was interrupted” and “the batch resumed without losing page 604 or capturing page 603 twice,” because in an archival context a dropped page is a permanent hole in the record and a duplicated page is a chain-of-custody defect that fails audit.

# Root-Cause Analysis of Mid-Batch Disconnects

A disconnect is not one event. It is a family of distinct hardware and transport failures that all surface to the host as “the device stopped answering,” and each leaves the capture in a subtly different state. Distinguishing them is what makes recovery deterministic instead of guesswork.

1. Firmware watchdog resets. Enterprise scanners run an internal watchdog that reboots the firmware if a capture thread stalls — a wedged ADF motor controller, a thermal event, or a garbage-collection pause in the on-board image processor. The reset clears the device’s volatile page counter and buffer, so on reconnect the scanner reports page 1 of a new session while the host believes it is mid-batch at page 604. Trusting the device counter after a watchdog reset is the single most common cause of duplicated pages.
2. USB bus re-enumeration. A power transient, a marginal cable, or an EMI event on the USB bus forces the host controller to re-enumerate the device. The kernel tears down the old device handle and issues a new one, invalidating every open file descriptor and in-flight bulk transfer. Any image buffer that had not been fully read from the endpoint is discarded by the driver — the page that was streaming at the moment of re-enumeration is lost, not merely delayed.
3. Network scanner TCP drops. For IP-attached devices, an idle-connection reset from an intervening firewall, a switch failover, or a device-side keep-alive timeout silently closes the socket. TCP guarantees ordered delivery of what was sent, so the hazard is not corruption but truncation: the frame in flight when the connection dropped arrives incomplete or not at all, and the next recv() on the stale socket raises ConnectionResetError rather than returning the remaining bytes.
4. Driver buffer loss. Even without a full disconnect, a firmware-to-host buffer overrun — the host falling behind the device’s capture rate — causes the driver ring buffer to overwrite unread frames. The device counter keeps advancing while pages quietly vanish from the host side, producing an off-by-N gap that is invisible until a page-count reconciliation catches it.

The unifying defect is treating the device’s notion of progress as authoritative. It is not durable across any of these failures. The host must maintain its own record of exactly which pages have been confirmed to durable storage, so that on reconnect it can reconcile the device’s volatile counter against a journal it trusts. The state machine below shows the capture loop moving page by page through a confirmed checkpoint, and the disconnect-and-resume path that re-enters the loop at the first unconfirmed page rather than at the device’s reported position.

The capture loop makes each page durable before the next begins; on disconnect it reconnects with bounded backoff, reads the journal for the last confirmed page, and resumes at the first unconfirmed page — never recapturing a confirmed page nor skipping an unconfirmed one.

# Step-by-Step Resolution: A Resumable, Journaled Capture Loop

Recovery rests on four mechanics working together: per-page capture checkpointing so progress is recorded at page granularity; a durable capture-progress journal that survives a host or device crash; idempotent resume that re-enters the loop at the first unconfirmed page; and reconciliation of the device’s on-board counter against the journal so a firmware reset cannot desynchronize the two. The order matters — a page is only ever marked confirmed after its bytes are fsynced and atomically renamed into place, so the journal can never claim a page that is not fully on disk.

The journal is an append-only newline-delimited JSON log. Each confirmed page appends one record; on startup the loop replays the log to find its high-water mark. Append-only is deliberate: an interrupted append leaves at most one truncated trailing line, which the loader skips, so recovery never reads a corrupt high-water mark.

from __future__ import annotations

import hashlib
import json
import logging
import os
import random
import time
from dataclasses import dataclass
from pathlib import Path
from typing import Callable, Optional

logger = logging.getLogger("archival.capture")


class ScannerDisconnected(Exception):
    """Raised when the device handle, USB endpoint, or TCP socket is lost mid-page."""


@dataclass(frozen=True)
class CaptureConfig:
    batch_id: str
    total_pages: int
    staging_dir: Path
    journal_path: Path
    max_reconnect_attempts: int = 8
    base_backoff_s: float = 0.5
    max_backoff_s: float = 30.0


def load_confirmed_pages(journal_path: Path) -> set[int]:
    """Replay the append-only journal, tolerating a truncated final line."""
    confirmed: set[int] = set()
    if not journal_path.exists():
        return confirmed
    with journal_path.open("r", encoding="utf-8") as handle:
        for line in handle:
            line = line.strip()
            if not line:
                continue
            try:
                record = json.loads(line)
            except json.JSONDecodeError:
                # A partial trailing append from an interrupted write — safe to skip.
                logger.warning("journal_line_truncated", extra={"raw": line[:80]})
                continue
            confirmed.add(int(record["page"]))
    return confirmed


def commit_page(cfg: CaptureConfig, page: int, payload: bytes) -> str:
    """Write a page durably, then journal it. Returns the recorded SHA-256."""
    digest = hashlib.sha256(payload).hexdigest()
    final_path = cfg.staging_dir / f"page.{page:05d}.tif"
    temp_path = final_path.with_suffix(".tif.part")

    # 1. Stream to a temp file and fsync so the bytes survive a crash.
    with temp_path.open("wb") as raster:
        raster.write(payload)
        raster.flush()
        os.fsync(raster.fileno())

    # 2. Atomic rename publishes the page; a half-written .part is never seen as final.
    os.replace(temp_path, final_path)

    # 3. Journal the page only after its bytes are durable and in place.
    record = {"batch_id": cfg.batch_id, "page": page, "sha256": digest}
    with cfg.journal_path.open("a", encoding="utf-8") as journal:
        journal.write(json.dumps(record) + "\n")
        journal.flush()
        os.fsync(journal.fileno())

    logger.info("page_confirmed", extra={"batch_id": cfg.batch_id, "page": page, "sha256": digest})
    return digest

The commit_page sequence is the checkpoint: fsync, atomic os.replace, then journal. Because the durable byte-write precedes the journal append, the journal is a strict subset of what is on disk — the loop can trust it without re-hashing every file on startup. This is the same fixity discipline the pipeline applies before promotion to archival storage, using SHA-256 as the audit-grade digest.

# Idempotent Resume and Bounded Reconnect

The capture loop skips any page already in the journal and treats capture as a retryable operation. When a ScannerDisconnected is raised, it reconnects with exponential backoff and jitter before retrying the same page — the identical backoff discipline described in Exponential Backoff with Jitter for Scanner Timeouts. Because a disconnected page was never journaled, retrying it cannot duplicate a confirmed page; because the loop iterates over the full page range and checks the journal, it cannot skip one.

def reconnect_with_backoff(
    cfg: CaptureConfig,
    connect: Callable[[], object],
    health_check: Callable[[object], bool],
) -> object:
    """Reconnect and confirm device health, bounded by max attempts and backoff."""
    last_error: Optional[Exception] = None
    for attempt in range(1, cfg.max_reconnect_attempts + 1):
        try:
            device = connect()
            if health_check(device):
                logger.info("scanner_reconnected", extra={"batch_id": cfg.batch_id, "attempt": attempt})
                return device
            raise ScannerDisconnected("health check failed after reconnect")
        except (ScannerDisconnected, ConnectionError, OSError) as exc:
            last_error = exc
            delay = min(cfg.max_backoff_s, cfg.base_backoff_s * (2 ** (attempt - 1)))
            delay = delay / 2 + random.uniform(0, delay / 2)  # full-ish jitter
            logger.warning(
                "reconnect_attempt_failed",
                extra={"batch_id": cfg.batch_id, "attempt": attempt, "sleep_s": round(delay, 3),
                       "error": str(exc)},
            )
            time.sleep(delay)
    raise ScannerDisconnected(f"exhausted reconnect attempts: {last_error}")


def run_resumable_capture(
    cfg: CaptureConfig,
    connect: Callable[[], object],
    capture_page: Callable[[object, int], bytes],
    health_check: Callable[[object], bool],
) -> dict[str, int]:
    """Capture every page exactly once, resuming across disconnects."""
    cfg.staging_dir.mkdir(parents=True, exist_ok=True)
    confirmed = load_confirmed_pages(cfg.journal_path)
    logger.info(
        "capture_resume_point",
        extra={"batch_id": cfg.batch_id, "already_confirmed": len(confirmed),
               "total_pages": cfg.total_pages},
    )

    device = reconnect_with_backoff(cfg, connect, health_check)
    captured = 0
    for page in range(1, cfg.total_pages + 1):
        if page in confirmed:
            continue  # idempotent skip — this page is already durable
        while True:
            try:
                payload = capture_page(device, page)
                commit_page(cfg, page, payload)
                captured += 1
                break
            except ScannerDisconnected as exc:
                logger.error("disconnect_mid_page", extra={"batch_id": cfg.batch_id,
                             "page": page, "error": str(exc)})
                device = reconnect_with_backoff(cfg, connect, health_check)
                # loop retries the SAME page; it was never journaled, so no duplicate

    result = {"newly_captured": captured, "confirmed_total": len(confirmed) + captured}
    logger.info("capture_complete", extra={"batch_id": cfg.batch_id, **result})
    return result

# Validation and Verification

Confirming the fix means proving three properties after a forced disconnect, not merely that the run finished. First, count reconciliation: the number of journal records must equal both the number of page.NNNNN.tif files in staging and the expected total_pages. A mismatch between the device’s on-board counter and the journal is expected after a watchdog reset — the reconciliation step deliberately discards the volatile device counter and trusts the journal.

def reconcile_batch(cfg: CaptureConfig, device_reported_count: int) -> dict[str, object]:
    """Verify journal, staging files, and device counter agree; the journal wins."""
    confirmed = load_confirmed_pages(cfg.journal_path)
    staged = {int(p.stem.split(".")[1]) for p in cfg.staging_dir.glob("page.*.tif")}
    missing = set(range(1, cfg.total_pages + 1)) - confirmed
    orphan_files = staged - confirmed  # bytes on disk but not journaled — must be re-committed
    counter_drift = device_reported_count - len(confirmed)
    report = {
        "confirmed": len(confirmed),
        "missing_pages": sorted(missing),
        "orphan_files": sorted(orphan_files),
        "device_counter_drift": counter_drift,
        "reconciled": not missing and not orphan_files,
    }
    log = logger.info if report["reconciled"] else logger.error
    log("batch_reconciliation", extra={"batch_id": cfg.batch_id, **report})
    return report

Second, idempotency under replay: re-running run_resumable_capture after a completed batch must capture zero new pages, because every page is already in the journal. Third, fixity re-derivation: re-hash each staged file and compare it to the sha256 recorded in the journal; a mismatch means a torn write slipped past os.replace, which should be impossible given the fsync-before-rename ordering but is worth asserting in audit. Each confirmed page should ultimately emit a PREMIS capture event carrying the recorded digest, folding the recovery into object provenance. This whole loop is the durable capture layer that Automating Batch Scanner Coordination with Python dispatches jobs into, and once a batch reconciles clean it is handed off to Async Task Queuing for Batches for downstream checksum verification and normalization.

# Edge Cases and Gotchas

Three archival-specific traps recur beyond the table. Multi-page TIFF containers complicate per-page checkpointing: if the batch writes one multi-page TIFF rather than one file per page, an interrupted append corrupts the whole container, so capture single-page masters and assemble the container only after all pages reconcile. Proprietary scanner “resume” extensions are seductive but untrustworthy — many vendor SDKs expose a resume_batch() call that replays from the device’s own volatile counter, which is exactly the counter a watchdog reset invalidated; treat the host journal as the sole source of truth and ignore device-side resume. Page-ordering metadata drift occurs when a resumed batch appends pages in capture order rather than sequence order; always key the staged filename on the logical page number (page.00604.tif), never on a monotonic write index, or a resume will scramble folio order. Malformed manifests and count mismatches surface downstream as failures against Batch Validation Schemas, so reconcile before dispatch.

For the underlying durability primitives, consult the Python os.fsync documentation and os.replace, which guarantee the atomic-commit behavior this recovery loop depends on.

# Frequently Asked Questions

# Why not just trust the scanner’s on-board page counter to resume?

Because that counter is volatile and does not survive the failures that cause mid-batch disconnects. A firmware watchdog reset clears it to zero, a USB re-enumeration hands back a fresh device with no memory of the session, and a driver buffer overrun advances it past pages that never reached the host. Resuming from the device counter after any of these produces duplicated or skipped pages. The host-side append-only journal is the only record that survives all four failure classes, so it — not the device — defines the resume point.

# How does the journal avoid marking a page confirmed before it is really on disk?

Ordering. commit_page writes the page to a temporary file, calls os.fsync to force the bytes to durable storage, atomically renames it into place with os.replace, and only then appends the journal record and fsyncs the journal. The journal is therefore always a subset of what is durably on disk. The one crash window — between the rename and the journal append — leaves an orphan file with no journal entry, which reconciliation detects and re-commits; it never leaves a journaled page whose bytes are missing.

# What stops a retry from capturing the same page twice?

A page is journaled only after it is fully committed, and the capture loop skips any page already in the journal. When a disconnect interrupts a page, that page was by definition never journaled, so retrying it after reconnect cannot create a duplicate. Confirmed pages are skipped on resume; unconfirmed pages are retried. This idempotent structure means a batch can be interrupted and resumed any number of times and still produce exactly one copy of each page.

# How many reconnect attempts should the backoff allow before giving up?

Enough to ride out a firmware reboot cycle (typically several seconds) but bounded so a genuinely dead device does not stall the batch forever. The example caps at eight attempts with exponential backoff and jitter, reaching a 30-second ceiling; that spans a normal watchdog reboot while still surfacing a hard failure within a couple of minutes. When attempts are exhausted, raise and let the operator intervene — the journal guarantees the batch can be resumed later from exactly where it stopped, so failing loudly costs nothing.

# Related

• Scanner API Integration & Routing — the parent stage: the scan-job contract and dispatch model this recovery loop plugs into.
• Automating Batch Scanner Coordination with Python — the device-control layer that dispatches jobs into this durable capture loop.
• Exponential Backoff with Jitter for Scanner Timeouts — the reconnect timing discipline this loop applies between disconnect and retry.