Syncing Local PRONOM Format Registry Snapshots for Reproducible, Air-Gapped Identification

Format identification is only trustworthy if it is reproducible: the same bytes fed to the same tool must resolve to the same PRONOM Unique Identifier (PUID) today, next quarter, and during an audit five years from now. That guarantee quietly dissolves the moment a pipeline lets DROID or Siegfried reach out to a live registry, or auto-updates its signature file whenever The National Archives publishes a new release. This walkthrough sits inside the Format Registry Integration area of the broader OAIS-Compliant Digital Preservation Architecture; it shows how to mirror the PRONOM signature files into an immutable, versioned local store, pin one snapshot per ingest run, and diff releases so that a change in the registry becomes a reviewed decision rather than an invisible drift in your provenance record.

# Why online-only format identification breaks reproducibility

Treating PRONOM as a live service reachable at identification time introduces three distinct failure modes, each of which undermines the deterministic behaviour that configuring format identification tools like DROID is supposed to deliver.

1. Online-only lookups cannot run air-gapped. Trustworthy digital repositories routinely process irreplaceable masters on segregated networks with no outbound internet. A tool configured to fetch the current signature file — or DROID’s built-in “check for signature updates” behaviour — simply fails, or worse, silently falls back to whatever stale file happens to be cached. Neither outcome is auditable, and neither can be replayed on demand.

2. Signature files change, and a change can shift a PUID. Each DROID release ships a binary signature file (DROID_SignatureFile_Vxxx.xml) and a container signature file (container-signature-YYYYMMDD.xml). New releases add signatures, tighten byte sequences, and re-prioritise matches. A byte stream that resolved to a generic fmt/ identifier under version 109 can resolve to a more specific PUID — or a different one entirely — under version 116. If your object records were written against the old file and re-identification runs against the new one, the two disagree and an auditor sees an unexplained mutation in the record.

3. Unpinned identification is non-deterministic by construction. When nothing records which signature version produced a result, identification becomes a function of wall-clock time: whatever release was current when the job happened to run. Two ingests of the same collection, weeks apart, can legitimately produce different PUIDs, and there is no artifact that explains the divergence. Determinism requires that identification be a pure function of the bytes and a fixed signature version — expressed compactly as $$\mathrm{PUID} = f(\text{bytes},\ v_{\text{sig}})$$ — which is impossible when $v_{\text{sig}}$ is implicit and floating.

The fix is to make $v_{\text{sig}}$ an explicit, immutable, checksummed artifact that lives beside the collection, and to bind every run to exactly one version. The diagram below shows the flow: upstream PRONOM feeds a versioned local snapshot store, and each ingest run pins one snapshot rather than reaching upstream.

Fetch the signature files once, store each release as an immutable checksummed snapshot, diff consecutive releases, and pin exactly one snapshot per ingest run so identification is deterministic and runs air-gapped.

# Building a versioned local snapshot store

The store is deliberately boring: one directory per signature version, each holding the raw XML plus a small JSON record of its provenance. The record captures the four facts an auditor needs to trust a snapshot — the version label, where it came from, when it was retrieved, and the SHA-256 digest of the exact bytes on disk. Fetching is the only step that touches the network, so it is wrapped in defensive error handling; a failed fetch returns None and logs a structured event rather than crashing the ingest that depends on it.

from __future__ import annotations

import hashlib
import json
import logging
import urllib.error
import urllib.request
from datetime import datetime, timezone
from pathlib import Path
from typing import Optional, TypedDict

logger = logging.getLogger("archival.pronom")


class SnapshotRecord(TypedDict):
    signature_version: str
    source_url: str
    retrieved_at: str
    sha256: str
    byte_length: int
    stored_path: str


def fetch_signature_snapshot(
    source_url: str,
    signature_version: str,
    snapshot_root: Path,
    timeout: float = 30.0,
) -> Optional[SnapshotRecord]:
    """Download one DROID signature file and store it as an immutable, checksummed snapshot.

    Returns the recorded provenance, or None if the network fetch failed.
    """
    snapshot_dir = snapshot_root / signature_version
    snapshot_dir.mkdir(parents=True, exist_ok=True)
    payload_path = snapshot_dir / "DROID_SignatureFile.xml"

    try:
        request = urllib.request.Request(
            source_url, headers={"User-Agent": "archival-ingest/1.0"}
        )
        with urllib.request.urlopen(request, timeout=timeout) as response:
            payload: bytes = response.read()
    except (urllib.error.URLError, TimeoutError, OSError) as exc:
        logger.error(
            "pronom_fetch_failed",
            extra={"source_url": source_url, "version": signature_version, "error": str(exc)},
        )
        return None

    digest = hashlib.sha256(payload).hexdigest()
    payload_path.write_bytes(payload)

    record: SnapshotRecord = {
        "signature_version": signature_version,
        "source_url": source_url,
        "retrieved_at": datetime.now(timezone.utc).isoformat(),
        "sha256": digest,
        "byte_length": len(payload),
        "stored_path": str(payload_path),
    }
    (snapshot_dir / "snapshot.json").write_text(json.dumps(record, indent=2, sort_keys=True))
    logger.info(
        "pronom_snapshot_recorded",
        extra={"version": signature_version, "sha256": digest, "bytes": len(payload)},
    )
    return record

Once written, a snapshot directory is treated as read-only. If a later release needs mirroring, it lands in a new directory under a new version label; existing snapshots are never overwritten. That immutability is what lets you re-run last quarter’s identification and get last quarter’s answer. The same fetch-and-record routine handles the container signature file — pass its URL and a version label such as container-20260706 — because container matching participates in the same PUID resolution and must be pinned alongside the binary signatures.

# Pinning a snapshot per ingest run

An ingest run declares the version it wants and resolves it through the store. Selection re-verifies fixity before handing the file to the identifier: it recomputes the SHA-256 of the bytes on disk and compares them to the digest recorded at retrieval, so silent corruption of the signature file is caught before it can taint an entire batch. This mirrors the fixity discipline used everywhere else in the pipeline, where SHA-256 is the audit-grade default and MD5 survives only as a legacy cross-check.

def select_pinned_snapshot(snapshot_root: Path, pinned_version: str) -> SnapshotRecord:
    """Resolve the snapshot pinned for a run and verify its bytes still match the recorded digest."""
    manifest_path = snapshot_root / pinned_version / "snapshot.json"
    if not manifest_path.is_file():
        logger.error("pinned_snapshot_missing", extra={"version": pinned_version})
        raise FileNotFoundError(f"No snapshot recorded for pinned version {pinned_version!r}")

    record: SnapshotRecord = json.loads(manifest_path.read_text())
    payload = Path(record["stored_path"]).read_bytes()
    actual = hashlib.sha256(payload).hexdigest()
    if actual != record["sha256"]:
        logger.error(
            "pinned_snapshot_corrupt",
            extra={"version": pinned_version, "expected": record["sha256"], "actual": actual},
        )
        raise ValueError(
            f"Snapshot {pinned_version} failed fixity: {actual} != {record['sha256']}"
        )

    logger.info("pinned_snapshot_selected", extra={"version": pinned_version, "sha256": actual})
    return record

The pinned version belongs in the run’s own manifest and, ultimately, in the object’s provenance. Recording “identified with DROID_SignatureFile_V116, sha256 c04d…7a” turns identification into a reproducible claim: anyone can retrieve that snapshot and re-derive the identical PUID. This is exactly the kind of technical provenance that feeds preservation-event records, and it is what keeps normalization decisions defensible when you later choose between profiles such as PDF/A-1b vs PDF/A-3 for format normalization.

# Diffing snapshots to see what changed

Before adopting a new release, compute the set of PUIDs each signature file can assign and diff them. A version that only adds PUIDs is low-risk; one that removes or re-scopes them demands a re-identification impact review against your holdings, because objects previously matched by a now-changed signature may resolve differently.

import xml.etree.ElementTree as ET
from typing import Set


def _format_puids(signature_xml: Path) -> Set[str]:
    """Extract the set of PUIDs a DROID signature file is able to assign."""
    puids: Set[str] = set()
    for element in ET.parse(signature_xml).iter():
        if element.tag.endswith("FileFormat"):
            puid = element.get("PUID")
            if puid:
                puids.add(puid)
    return puids


def diff_snapshots(old_xml: Path, new_xml: Path) -> dict[str, list[str]]:
    """Report PUIDs added or removed between two signature-file snapshots."""
    old_puids = _format_puids(old_xml)
    new_puids = _format_puids(new_xml)
    delta = {
        "added": sorted(new_puids - old_puids),
        "removed": sorted(old_puids - new_puids),
    }
    logger.info(
        "snapshot_diff_computed",
        extra={"added": len(delta["added"]), "removed": len(delta["removed"])},
    )
    return delta

A non-empty removed set is the trigger to run the new snapshot against a sample of already-identified objects and confirm no PUID silently flips. Where two candidate signatures genuinely compete for the same byte stream, that is a distinct problem handled when resolving DROID format identification conflicts; the diff only tells you which formats to scrutinise.

# Validation and verification

Confirm the snapshot machinery works before it guards a real ingest, using evidence rather than trust in configuration:

• Assert byte-for-byte fixity. After fetch_signature_snapshot, recompute the SHA-256 of the stored XML and assert it equals the sha256 field in snapshot.json. A mismatch means the file was truncated or altered after retrieval and the snapshot must be re-fetched.
• Prove determinism across runs. Identify a fixed corpus twice against the same pinned snapshot; every PUID must be identical. Then identify it against two different snapshots and confirm any divergence is fully explained by the diff_snapshots output — no unexplained changes.
• Prove the air-gap. Disconnect the network and run select_pinned_snapshot followed by identification end to end. It must succeed entirely from local bytes; any outbound call is a leak that reintroduces the online dependency you removed.
• Audit the provenance, not the log line. The pinned version and its digest belong in a durable preservation event, so identification can be reconstructed from the record long after the log rotates.

# Edge cases and gotchas

Three archival-specific traps recur. First, version labels must be stable and meaningful: derive them from the upstream release identifier (for example V116), not from a local timestamp, or two mirrors of the same release end up with different labels and the diff becomes meaningless. Second, the binary and container signature files version independently — a new container file can ship without a new binary file, so track and pin them separately even when they are usually fetched together. Third, Siegfried compiles its own signature from the PRONOM release, so if your repository runs both DROID and Siegfried, snapshot the source release and record which compiled artifact each tool consumed; pinning the upstream version is necessary but, for Siegfried, not sufficient without also pinning its built signature.

# Frequently Asked Questions

# Why not just let DROID auto-update its signature file?

Auto-update makes identification depend on whatever release happens to be current when a job runs, which is precisely the non-determinism that breaks reproducibility and audit. It also requires outbound network access that segregated preservation environments do not permit. Pinning an immutable local snapshot gives you a fixed, replayable input; you still adopt new releases, but as a reviewed, logged decision rather than a silent side effect of running the tool.

# How do I choose which signature version to pin for a run?

Default to the newest snapshot that has passed a re-identification impact review, and keep it stable for the duration of a collection or campaign so all objects in that campaign share one identification basis. Record the pinned version in the run manifest. When a newer release lands, diff it against the current pin, review any removed or re-scoped PUIDs against your holdings, and only then promote it to the default for future runs.

# Do I need to snapshot the container signature file separately?

Yes. DROID resolves many formats — OOXML, ODF, and other ZIP- or OLE2-based containers — using the container signature file, which versions on its own schedule from the binary signature file. Pinning only the binary file leaves container identification floating, so mirror both, store each with its own checksum and retrieval date, and pin the matching pair for every run.

# What happens if a snapshot fails its fixity check at run time?

select_pinned_snapshot raises before any object is identified, which fails the run fast instead of stamping a whole batch with a PUID derived from corrupted signatures. Recovery is to re-fetch that version into a fresh directory, confirm the new bytes match the digest you recorded upstream, and re-run. Because snapshots are immutable and checksummed, you always have the original digest to verify against.

# Related

• Format Registry Integration — the parent area: how PRONOM, PUIDs, and registry mirroring underpin every normalization decision.
• Configuring Format Identification Tools like DROID — pointing DROID and Siegfried at a pinned local signature file and disabling runtime auto-update.
• Resolving DROID Format Identification Conflicts — arbitrating when multiple signatures in a pinned snapshot compete for the same byte stream.