Automating Batch Scanner Coordination with Python: Resolving Edge Cases in Archival Digitization

Coordinating high-throughput batch scanners in archival digitization environments requires deterministic control over hardware states, network I/O, and downstream preservation handoffs. This page is the edge-case troubleshooting reference for the device-control layer specified in Scanner API Integration & Routing: it takes the scan-job contract and dispatch model defined there and shows what to do when a device is nominally ready but the coordinator still corrupts, drops, or misroutes a master. When institutional programs scale beyond single-operator workstations inside the broader Automated Ingestion & Batch Scanning Workflows pipeline, Python becomes the orchestration layer bridging proprietary scanner SDKs with long-term preservation infrastructure, and three failure modes come to dominate: a firmware READY flag that fires before the image buffer has drained, a worker pool that deadlocks behind a single jammed feeder, and concurrent writes that publish half-flushed TIFFs to staging. The specific problem solved here is the gap between “the scanner reported it was done” and “the master is complete, verified, and safely committed.”

# Root-Cause Analysis of Coordination Failures

Coordination failures rarely stem from bad hardware. They originate in the seam between firmware state and host-side I/O, where an optimistic control script trusts a signal that does not mean what it appears to mean. Three primary causes dominate production capture farms.

1. The READY signal is a mechanical acknowledgment, not a data-transfer completion flag. Enterprise document scanners raise READY when the automatic document feeder (ADF) has cleared the page path, but the internal image buffer may still be flushing to the host. TCP guarantees ordered, lossless delivery, so the failure is never packet loss — it is application-level framing: a single socket.recv() returns whatever bytes have arrived so far, not a complete message. A coordinator that reads immediately after polling the READY register can observe a partial payload, or a status frame interleaved with image bytes still draining from the firmware buffer.
2. Unbounded worker pools deadlock behind a single stalled device. When a jammed ADF or a hung firmware garbage-collection pause stops responding, a naive thread-per-job pool blocks every worker waiting on that socket, and the whole batch stalls. Uncompressed 600 DPI archival masters exceed 100 MB per page — and rise well beyond that for large-format or high-bit-depth captures — so a backlog of in-flight buffers also drives the workers into memory exhaustion.
3. Non-atomic routing publishes partial masters. When concurrent workers write directly into a shared staging namespace, a downstream consumer can read a multi-page TIFF that is still being appended, or a metadata sidecar can detach from its raster file. The result looks like a schema violation to Batch Validation Schemas but is really a write-ordering artifact.

Robust clients therefore read until a known length or delimiter, confirm the firmware-to-host buffer has drained before dispatching the next job, and publish only through an atomic move. The sequence below shows the coordinator triggering a capture, verifying the buffer flush before trusting the data, and writing to staging only after the flush is confirmed.

The coordinator dispatches the next job only after confirming both READY state and a drained socket buffer.

# Step 1: Confirm the Buffer Flush Before Trusting the Data

The fix for the READY-versus-flushed race is a two-stage acknowledgment loop. First assert the firmware state; then confirm the receive buffer has actually drained before the coordinator treats the capture as complete and dispatches the next job. Polling READY alone is exactly the check that fails.

import asyncio
import logging
import select
import socket
from enum import Enum

logger = logging.getLogger("scanner.ack")


class ScannerState(Enum):
    IDLE = "IDLE"
    FEEDING = "FEEDING"
    CAPTURING = "CAPTURING"
    FLUSHING = "FLUSHING"
    READY = "READY"


class HardwareAckLoop:
    def __init__(self, host: str, port: int, timeout: float = 5.0) -> None:
        self.host = host
        self.port = port
        self.timeout = timeout

    async def verify_buffer_flush(self) -> bool:
        """Poll firmware state, then confirm the TCP receive buffer has drained."""
        loop = asyncio.get_running_loop()
        with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
            sock.settimeout(self.timeout)
            sock.connect((self.host, self.port))

            # Vendor-specific status opcode; returns the current firmware state frame.
            sock.sendall(b"\x01\x05\x00\x00")
            raw = await loop.run_in_executor(None, sock.recv, 1024)
            current_state = raw.decode("utf-8", errors="replace").strip()

            if current_state != ScannerState.READY.value:
                logger.info("flush_wait host=%s state=%s", self.host, current_state)
                return False

            # A socket that still reports readable after READY means image bytes are
            # still arriving: the firmware-to-host flush is incomplete and the buffer
            # is not safe for the next dispatch.
            has_pending = await loop.run_in_executor(None, self._has_pending_bytes, sock)
            if has_pending:
                logger.warning("flush_incomplete host=%s residual_bytes=True", self.host)
            return not has_pending

    def _has_pending_bytes(self, sock: socket.socket) -> bool:
        """select reports the socket readable while unread bytes remain in the buffer."""
        readable, _, _ = select.select([sock], [], [], 0.1)
        return bool(readable)

Reading until the buffer is provably empty — rather than reading once and trusting READY — is what eliminates the fragmented and interleaved payloads that otherwise reach staging.

# Step 2: Isolate a Jammed Device Without Stalling the Batch

To keep one stalled feeder from cascading into a full pipeline halt, the coordinator must decouple hardware polling from image processing and guard every device call behind a circuit breaker. Long-running capture cycles run asynchronously against Async Task Queuing for Batches, which fans the completed masters out to checksum and transfer workers; the breaker is what stops the coordinator from repeatedly dispatching work to a device that is already failing. Size the worker pool to one active capture thread per physical device, reserving additional threads for verification and network transfer, and set an explicit memory ceiling that accommodates the largest expected master.

import logging
import time
from dataclasses import dataclass
from typing import Any, Awaitable

logger = logging.getLogger("scanner.breaker")


@dataclass
class CircuitBreaker:
    failure_threshold: int = 3
    recovery_timeout: float = 60.0
    _failures: int = 0
    _last_failure_time: float = 0.0
    _state: str = "CLOSED"

    async def execute(self, coro: Awaitable[Any]) -> Any:
        if self._state == "OPEN":
            if time.time() - self._last_failure_time >= self.recovery_timeout:
                self._state = "HALF-OPEN"
                logger.info("breaker_probe state=HALF-OPEN")
            else:
                raise RuntimeError("Circuit breaker OPEN: scanner offline or ADF jammed.")

        try:
            result = await coro
            self._reset()
            return result
        except Exception:
            self._record_failure()
            raise

    def _record_failure(self) -> None:
        self._failures += 1
        self._last_failure_time = time.time()
        if self._failures >= self.failure_threshold:
            self._state = "OPEN"
            logger.warning("breaker_open failures=%d", self._failures)

    def _reset(self) -> None:
        if self._state != "CLOSED":
            logger.info("breaker_closed recovered=True")
        self._failures = 0
        self._state = "CLOSED"

A tripped breaker removes the device from the eligible dispatch pool until a single probe in HALF-OPEN confirms it is healthy, so transient firmware hangs never propagate into a batch-wide stall.

# Step 3: Route to Staging With an Atomic Commit

Once a master is captured and its buffer verified, it must be published to staging in a way no downstream consumer can observe mid-write. Writing into a temporary path and then renaming into the final namespace makes the transition atomic: a reader sees the whole file or nothing. This is the write-ordering guarantee that keeps concurrent workers from producing the orphaned derivatives and split multi-page TIFFs described above.

import logging
import os
from pathlib import Path

logger = logging.getLogger("scanner.routing")


def commit_master(temp_path: Path, staging_root: Path, batch_id: str, page: int) -> Path:
    """Atomically publish a captured master so downstream consumers never see a partial file."""
    target_dir = staging_root / batch_id
    target_dir.mkdir(parents=True, exist_ok=True)
    final_path = target_dir / f"{batch_id}_p{page:04d}.tif"

    # os.replace is atomic when source and target share a filesystem, so a downstream
    # worker either sees the whole master or nothing -- never a truncated read.
    os.replace(temp_path, final_path)
    logger.info("master_committed batch=%s page=%d path=%s", batch_id, page, final_path)
    return final_path

Keep the temporary file on the same filesystem as the staging root; a cross-device os.replace falls back to a non-atomic copy-then-delete and reintroduces the partial-read window. From staging, validated payloads transition into OCR and metadata stages, where enrichment via Metadata Extraction Workflows embeds standardized XMP sidecars and preserves the original capture parameters. Faults that are transient — a dropped socket, a momentary timeout — should route to Error Handling & Retry Logic for exponential backoff, while a persistent checksum mismatch or a malformed TIFF header triggers immediate quarantine and an operator alert.

# Validation and Verification

A coordination fix is only trustworthy once you can prove the master arrived intact and record that proof. Three checks confirm the pipeline behaved, and the third is what makes the verdict defensible under audit.

• Capture and correlate the wire traffic. Configure tcpdump or Wireshark to filter on the scanner’s management VLAN, isolating control-plane traffic from bulk image streams, and cross-reference packet timestamps against the Python logging output. Latency spikes that line up with a flush_incomplete warning confirm the buffer race is being caught rather than masked; spikes with no corresponding retry point to firmware garbage-collection pauses that need a longer recovery timeout.
• Assert fixity end to end. Compute the SHA-256 of the committed master and confirm it matches the digest the scanner reported at capture time. A buffer-flush fix is only proven when the bytes on disk are bit-identical to what the device produced.
• Emit a PREMIS capture event. Record every dispatch, flush verification, and commit as a preservation event following PREMIS Metadata Mapping, so the audit trail captures which device produced the object and that its buffer was verified drained. This is the alignment point with the OAIS Reference Model Implementation: a capture event populates the Provenance information every package must carry to become a legitimate AIP. Compliance also requires validating bit-depth, color space, DPI consistency, and embedded metadata against recognized imaging metrics such as the ISO 19264-1 imaging-quality standard before assets are committed to the repository.

# Edge Cases and Gotchas

• Multi-page TIFF fragmentation under concurrency. When a scanner packager appends pages incrementally, a manifest can list all pages while the byte payloads are still flushing. Gate the commit on a capture-complete sentinel from the device, not a fixed sleep, or an atomic rename will still publish an incomplete master.
• Proprietary scanner opcode extensions. Vendor status registers rarely follow a shared convention: the same READY bit can mean “page cleared” on one firmware and “buffer drained” on another. Pin the status opcode and its interpretation per device_class rather than assuming the fleet is homogeneous.
• select false-readable after a half-close. If the scanner half-closes the connection, select reports the socket readable even though recv will return zero bytes. Distinguish a genuine residual-byte condition from an EOF by checking the length of the next non-blocking read before deciding the flush is incomplete.
• Latin-1 status frames on legacy devices. Older controllers emit status text as ISO-8859-1, so a strict decode("utf-8") raises mid-poll and looks like a hardware fault. Decode status frames with errors="replace" and validate against the enumerated ScannerState values instead of trusting the raw string.

By treating scanner coordination as a deterministic state machine — verify the flush, guard each device with a breaker, commit atomically, then record the event — archival engineering teams eliminate race conditions and misrouting instead of papering over them with fixed delays, turning fragile hardware dependencies into resilient, auditable preservation pipelines.

# Related

• Scanner API Integration & Routing — the parent stage: the scan-job contract, device-selection routing, and the health model this page troubleshoots.
• Async Task Queuing for Batches — where verified masters fan out asynchronously to checksum and transfer workers.
• Error Handling & Retry Logic — where a transient dispatch fault goes: backoff, quarantine, and re-capture scheduling.