Detecting Schema Drift Across Scanner Firmware Versions: Catching Silent Manifest Shape Changes Before Ingest

A scanner firmware update never announces that it renamed resolution_dpi to resolution and started reporting the value in pixels-per-inch instead of dots-per-inch. It just ships, the overnight batch runs, and the next morning either half the run is stuck in the validator or — far worse — every object sailed through with a resolution field that now means something subtly different. This page is the drift-detection reference for the Batch Validation Schemas stage: it takes the versioned profiles defined there and shows how to catch the moment a device’s manifest shape diverges from what the profile expects, before a full batch commits corrupt or misaligned metadata into the archive. The specific failure it solves is the gap between “the batch validated” and “the batch validated against the schema we actually meant” — the ambiguity that turns a routine firmware bump into a silent provenance defect discovered months later during an audit.

# Root-Cause Analysis of Firmware-Induced Schema Drift

Schema drift is distinct from a malformed manifest. The bytes are well-formed JSON or XMP, the file parses cleanly, and often the batch even passes a permissive validator — yet the meaning has shifted because the device that produced it changed. Four drift classes account for nearly every case seen when a scanner fleet spans mixed firmware revisions:

1. Renamed or relocated keys. A firmware revision renames scanner_model to device.model or moves color_profile from the manifest root into a nested capture object. Any validator keyed on the old flat path now reports a missing required field, even though the data is present one level down. When two firmware versions run side by side in the same fleet, the batch queue interleaves both shapes and the failures look intermittent rather than systematic.
2. Unit changes with an unchanged key. The most dangerous class, because nothing fails. A value keyed resolution_dpi continues to validate as an integer in range, but the firmware now populates it from a sensor reporting pixels-per-inch, or switches a file-size field from bytes to kilobytes. The type is stable, the range check passes, and the wrong number is recorded as fact. This is the drift that only a unit-aware comparison — not a JSON Schema type check — will catch.
3. Encoding and tag-encoding shifts. EXIF and XMP blocks emitted by an updated firmware may switch a rational-encoded resolution to a plain integer, change a UTF-8 string to a Latin-1 byte sequence, or alter the container from Exif.Image.XResolution to a vendor MakerNote tag. The manifest still lists a resolution; the decoder now reads it differently.
4. New nesting and added keys. A firmware adds a warranty or calibration object, or wraps existing fields in a new envelope. Additive changes are usually benign, but a strict additionalProperties: false profile will reject them outright, and a lenient profile will accept a re-nested payload whose required fields are now unreachable at their old paths.

The common thread is that a per-document validator sees each object in isolation and cannot tell “this field is missing” from “this field moved” or “this number changed units.” Drift detection operates one level up: it compares the shape of a sampled batch against the expected shape for its declared firmware version, and classifies the difference before a single object is promoted. That classification — cosmetic, additive, or breaking — is what decides whether the batch proceeds, waits for a human, or is quarantined.

Sample a canary slice, infer its shape, diff against the expected profile for the declared firmware, then route: cosmetic drift proceeds, additive drift holds for review, breaking drift fails closed to quarantine.

# Step-by-Step Resolution: Infer, Diff, and Route Before Full Ingest

The fix has one non-negotiable precondition: capture the firmware and model version on every batch. Drift is meaningless without a version to attribute it to, and the version is what keys the expected profile. Have Scanner API Integration & Routing stamp firmware_version and scanner_model into the batch manifest at capture time so the detector never has to guess which shape it should expect.

The detector itself is three functions: infer a shape signature from a manifest, diff two signatures, and classify the result. The signature flattens the manifest into dotted key paths, each carrying a Python type and — critically — a declared unit derived from the key name, so a dpi→ppi swap is visible even when the type never changes.

from __future__ import annotations

import logging
from dataclasses import dataclass, field
from enum import Enum
from typing import Any

logger = logging.getLogger("archival.drift")


class DriftSeverity(str, Enum):
    NONE = "none"
    ADDITIVE = "additive"
    BREAKING = "breaking"


@dataclass(frozen=True)
class FieldShape:
    """Type and unit signature for a single flattened manifest key path."""

    path: str
    py_type: str
    unit: str | None = None


# Key-name tokens that imply a measurement unit; a change here is silent unit drift.
_UNIT_TOKENS: dict[str, str] = {
    "dpi": "dpi",
    "ppi": "ppi",
    "resolution": "dpi",
    "filesize": "bytes",
    "bytes": "bytes",
}


def _unit_hint(key: str) -> str | None:
    """Infer the declared measurement unit implied by a key name."""
    lowered = key.lower()
    for token, unit in _UNIT_TOKENS.items():
        if token in lowered:
            return unit
    return None


def infer_signature(manifest: dict[str, Any], prefix: str = "") -> dict[str, FieldShape]:
    """Flatten a manifest into {dotted.path: FieldShape}, recursing into nested objects."""
    shape: dict[str, FieldShape] = {}
    for key, value in manifest.items():
        path = f"{prefix}{key}"
        if isinstance(value, dict):
            shape.update(infer_signature(value, prefix=f"{path}."))
            continue
        shape[path] = FieldShape(path=path, py_type=type(value).__name__, unit=_unit_hint(key))
    return shape

The diff compares an observed signature against the expected profile and produces a structured report. Missing keys, type changes, and unit changes are treated as breaking — they either fail validation or, worse, pass with wrong data. Added keys are additive and warrant review rather than rejection. The report is keyed by firmware version so alerting can attribute every drift event to the revision that introduced it, and it is emitted through structured logging for the metrics pipeline.

@dataclass
class DriftReport:
    """Structured drift verdict for one batch, attributed to its firmware version."""

    firmware_version: str
    severity: DriftSeverity = DriftSeverity.NONE
    missing_keys: list[str] = field(default_factory=list)
    added_keys: list[str] = field(default_factory=list)
    type_changes: list[str] = field(default_factory=list)
    unit_changes: list[str] = field(default_factory=list)


def _classify(report: DriftReport) -> DriftSeverity:
    """Breaking drift removes, retypes, or re-units a field; added-only drift is additive."""
    if report.missing_keys or report.type_changes or report.unit_changes:
        return DriftSeverity.BREAKING
    if report.added_keys:
        return DriftSeverity.ADDITIVE
    return DriftSeverity.NONE


def diff_signature(
    observed: dict[str, FieldShape],
    expected: dict[str, FieldShape],
    firmware_version: str,
) -> DriftReport:
    """Diff an observed manifest shape against the expected profile for a firmware version."""
    report = DriftReport(firmware_version=firmware_version)
    report.missing_keys = sorted(expected.keys() - observed.keys())
    report.added_keys = sorted(observed.keys() - expected.keys())
    for path in sorted(observed.keys() & expected.keys()):
        obs, exp = observed[path], expected[path]
        if obs.py_type != exp.py_type:
            report.type_changes.append(f"{path}: {exp.py_type}->{obs.py_type}")
        if obs.unit != exp.unit:
            report.unit_changes.append(f"{path}: {exp.unit}->{obs.unit}")
    report.severity = _classify(report)
    logger.info(
        "schema_drift_report",
        extra={
            "firmware_version": firmware_version,
            "severity": report.severity.value,
            "missing_keys": report.missing_keys,
            "added_keys": report.added_keys,
            "type_changes": report.type_changes,
            "unit_changes": report.unit_changes,
        },
    )
    return report

Finally, the canary gate ties it together. Rather than validating the whole batch first, it shadow-validates a small sample — the canary slice from the drift-detection pipeline above — and returns a routing decision. Breaking drift fails closed to quarantine; additive drift holds for a human; a clean or cosmetic result releases the full batch for ingest.

def evaluate_canary(
    sample_manifest: dict[str, Any],
    expected: dict[str, FieldShape],
    firmware_version: str,
) -> bool:
    """Shadow-validate one sampled manifest; return True only when full ingest is safe."""
    observed = infer_signature(sample_manifest)
    report = diff_signature(observed, expected, firmware_version)
    if report.severity is DriftSeverity.BREAKING:
        logger.error(
            "drift_quarantine",
            extra={
                "firmware_version": firmware_version,
                "missing_keys": report.missing_keys,
                "type_changes": report.type_changes,
                "unit_changes": report.unit_changes,
            },
        )
        return False
    if report.severity is DriftSeverity.ADDITIVE:
        logger.warning(
            "drift_hold_for_review",
            extra={"firmware_version": firmware_version, "added_keys": report.added_keys},
        )
        return False
    return True


if __name__ == "__main__":
    logging.basicConfig(level=logging.INFO)
    # Expected profile derived once from a known-good golden manifest.
    golden = {
        "scanner_model": "OpticPro A3",
        "firmware_version": "3.4.1",
        "resolution_dpi": 600,
        "capture": {"color_profile": "AdobeRGB"},
    }
    expected_profile = infer_signature(golden)

    # A batch from firmware 3.5.0 renamed resolution_dpi -> resolution (now ppi).
    drifted = {
        "scanner_model": "OpticPro A3",
        "firmware_version": "3.5.0",
        "resolution": 600,
        "capture": {"color_profile": "AdobeRGB"},
    }
    safe = evaluate_canary(drifted, expected_profile, firmware_version="3.5.0")
    logger.info("canary_decision", extra={"proceed": safe})

# Validation and Verification

Confirm the detector works by proving it catches drift it was designed to catch, not by trusting that it ran:

• Replay a known-good golden batch. Feed a captured golden manifest back through evaluate_canary against its own profile; the report severity must be none. A false-positive on the golden manifest means the unit-hint or flattening logic is over-eager and will quarantine healthy batches.
• Inject each drift class. Rename a key, change a type, and swap resolution_dpi for a resolution field, then assert the report lists exactly the expected missing_keys, type_changes, and unit_changes. The unit-swap case is the one to guard hardest — it is the only failure that a plain JSON Schema type check will not catch.
• Track a drift rate and alert on it. Emit each severity to your metrics backend and alert when the breaking-drift rate over a rolling window crosses a threshold. For $n$ sampled batches with $b$ classified breaking, the drift rate is

$$ D = \frac{b}{n}, \qquad \text{alert when } D > D_{\text{max}} $$

A single breaking batch from a new firmware version should trip the alert; a steady non-zero $D$ across an established version signals a profile that has fallen behind the fleet and needs re-baselining.

• Confirm fail-closed behavior end to end. Point a breaking batch at the gate and verify it lands in quarantine rather than in archival storage — and that the per-document schema compliance check never runs on the un-promoted objects. Fail-closed means the absence of a positive “safe” verdict blocks ingest, so a detector crash quarantines rather than releases.

# Edge Cases and Gotchas

Beyond the table, three archival-specific traps recur:

• Mixed-firmware batches. A single overnight run can interleave objects from scanners on different firmware. Sample and profile per scanner_model + firmware_version, never per batch — otherwise one drifted device is averaged away and its objects slip through. Route the version key from the manifest, not from a fleet-wide assumption.
• Cosmetic vs. semantic drift. Whitespace, key ordering, and a redundant "schemaVersion" bump are cosmetic and must not quarantine a batch, or operators will learn to ignore the alerts. Keep the unit and type rules strict and let ordering and formatting differences fall through as none.
• Profile staleness. The expected profile is only as good as the golden manifest it was baselined from. When a firmware version is deliberately adopted fleet-wide, re-baseline the profile from a reviewed golden batch of that version and record the change — otherwise every batch from the new-but-approved firmware reads as breaking drift and the quarantine fills with healthy content.

Drift detection is a gate, not a fixer: it decides whether a batch is safe to ingest under the profile it claims to match. Pair it with the versioned profiles it reads and it turns a silent firmware change from a months-later audit finding into a same-night quarantine event. For the flattening and set logic above, the Python dataclasses documentation is the relevant reference.

# Frequently Asked Questions

# How is schema drift different from a normal validation failure?

A validation failure means the manifest is malformed against a fixed schema — a missing required field, a wrong type, an out-of-range value. Drift means the manifest is well-formed but its shape or meaning changed relative to what the profile was built for, usually because the device that produced it was updated. The dangerous drift cases pass ordinary validation: a resolution field that switched from dpi to ppi is still an integer in range. Detecting drift requires comparing a batch’s inferred shape to the expected shape for its firmware version, not just checking each field against a static rule.

# Why sample a canary batch instead of validating everything?

Because the point is to decide whether the whole batch is safe before committing any of it. A canary slice of a few documents is enough to reveal a firmware-wide shape change — every object from the same device shares the same manifest shape — and it fails fast: a breaking result quarantines the run in seconds instead of after thousands of objects have already been promoted with wrong metadata. Full per-document validation still runs, but only on batches the canary has cleared.

# How do I catch a unit change when the field type never changes?

Derive a declared unit from the key name and compare units, not just types. A field keyed resolution_dpi implies dpi; if a firmware renames it resolution the unit hint changes even though both values are integers, and the diff flags a unit change. This is the only signal that separates a genuine 600-dpi value from a 600-ppi value carrying the same Python type, which is why unit comparison is a first-class part of the shape signature rather than an afterthought.

# What should happen to a batch classified as breaking drift?

It should fail closed to a quarantine queue, not proceed and not silently retry. Fail-closed means ingest is blocked unless the detector returns a positive “safe” verdict, so even a detector crash quarantines rather than releases. From quarantine, an operator reviews the drift report, decides whether the firmware change is legitimate, and either re-baselines the expected profile for the new version or rejects the batch back to the scanning team.

# Related

• Versioning JSON Schema Profiles for Ingest Batches — how the per-firmware expected profiles this detector diffs against are versioned and pinned.
• Validating Schema Compliance During Digital Ingest — the per-document validator that runs on batches the drift gate has cleared.
• Scanner API Integration & Routing — where firmware and model versions are stamped into the manifest at capture time.