Format Registry Integration in OAIS-Compliant Digital Preservation Architecture

Format registry integration is the intelligence layer that turns a raw bitstream into an actionable preservation object. Within the parent OAIS-Compliant Digital Preservation Architecture, this is the subsystem that resolves every ingested file to an authoritative, versioned format identity, scores it against institutional risk thresholds, and emits the technical metadata that downstream stages depend on. It sits directly between signature-based Preservation Format Identification — which tells you what a file is from its bytes — and PREMIS Metadata Mapping, which records what you did about it. For archivists and Python automation engineers, the registry is not a static lookup table; it is a continuously synchronised source of truth for migration triggers, obsolescence monitoring, and provenance. This page specifies the normalised registry-entry contract, the synchronisation control flow, a production-ready polling and mapping implementation, and the compliance rules every registry lookup must satisfy before an object is promoted to archival storage.

# Registry Entry Specification and Normalisation Contract

Authoritative registries do not agree on a schema. PRONOM exposes PUIDs, binary signatures, and byte offsets through its XML signature files; the Library of Congress Sustainability of Digital Formats resource publishes narrative sustainability factors; Wikidata carries loosely typed format entities. Before any of these can drive a preservation decision, the integration layer must normalise them into a single, version-pinned internal contract. Modelling that contract as an immutable, validated data structure gives every worker in the pipeline the same deterministic view of a format, and lets a malformed registry response be rejected before it can poison a metadata record.

The field specification below is the authoritative shape every resolved registry entry must honour. A response that fails any constraint is quarantined at the resolution boundary and never reaches PREMIS.

The two fields that carry the most preservation weight are registry_version and response_hash. Pinning the signature-file release the entry was resolved against is what makes an identification reproducible: when PRONOM ships a new signature file that reassigns a PUID for a legacy format, an unpinned pipeline silently rewrites its own historical baseline, while a pinned one surfaces the drift as a diff. The response_hash — a digest over the canonical serialisation of the raw registry payload — is what lets an auditor confirm, months later, that the format identity recorded in an Archival Information Package is byte-for-byte the one the registry actually returned.

# OAIS Functional Alignment and Proactive Planning

The integration directly operationalises the Preservation Planning and Ingest functional entities defined in the code-driven OAIS Reference Model implementation. By continuously querying authoritative registries, the pipeline evaluates format viability against institutional risk thresholds at ingest time rather than during a periodic audit. Without this automated linkage, preservation planning stays reactive: obsolescence is discovered only when a file fails to render, by which point the originating hardware and software context may be gone.

Risk is rarely a single registry field. A defensible model composes several signals — registry sustainability score, adoption trend, and whether an open-source renderer exists — into one comparable number. Expressed as a weighted sum, the composite format-risk score $R$ for a format $f$ is:

$$ R(f) = w_{s},S(f) + w_{a},A(f) + w_{r},\big(1 - \mathrm{Ren}(f)\big) $$

where $S(f)$ is the normalised sustainability penalty, $A(f)$ the adoption-decline factor, $\mathrm{Ren}(f) \in {0, 1}$ flags whether an open renderer exists, and the weights satisfy $w_{s} + w_{a} + w_{r} = 1$. A format whose $R(f)$ crosses the migration threshold becomes a Preservation Planning trigger, feeding a normalisation queue rather than waiting for a human to notice the format is dying.

# Deterministic Resolution Flow

The flow below outlines the deterministic path a file follows from signature-based identification through registry resolution to PREMIS metadata and an audit record.

A file is identified by binary signature, resolved to a PRONOM PUID, mapped into PREMIS format designation fields, then persisted with an audit trail.

Signature matching itself relies on binary offsets, container parsing, and heuristic analysis handled by open-source tools such as Siegfried, DROID, and Apache Tika. Institutional collections, however, routinely contain proprietary, legacy, or highly specialised file types that fall outside the standard signature sets — the mechanics of extending coverage for those are covered in depth in configuring format identification tools like DROID. Custom signatures for rare formats are compiled into version-controlled signature files, validated against a known test corpus, and deployed through CI/CD so identification stays consistent across every distributed ingest node.

# Continuous Registry Synchronisation and CI/CD Deployment

The velocity of digital format evolution demands continuous synchronisation between external registries and internal systems. A deterministic update pipeline fetches registry deltas, validates XML/JSON schema compliance, and atomically swaps the local signature database only after the new definitions pass regression testing against a golden corpus. Atomicity matters: a half-applied signature update is worse than a stale one, because it can assign PUIDs from two different registry releases within the same batch. Every promotion is gated on the new definitions producing zero false positives and zero regressions against the reference set, and the swap must coordinate with the multi-node replication that Long-Term Storage Architecture manages so no two archival nodes ever resolve the same file against different registry versions.

# Core Implementation: Deterministic Registry Resolution

The module below is a production-ready resolution pattern. It polls an external registry mirror, canonicalises and hashes the raw response for auditability, validates the payload against the registry-entry contract, and maps the result into PREMIS-compliant technical metadata. It uses only the standard library so it runs unchanged inside a locked-down ingest worker, and every failure path is classified and logged rather than swallowed.

"""
format_registry_integration.py
Deterministic OAIS format registry resolution with audit logging and PREMIS mapping.
Requires: Python 3.9+
"""

import hashlib
import json
import logging
import urllib.error
import urllib.request
from dataclasses import dataclass
from datetime import datetime, timezone
from typing import Any, Dict, Optional

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s | %(levelname)s | %(message)s",
    datefmt="%Y-%m-%dT%H:%M:%SZ",
)
logger = logging.getLogger("oais_format_registry")

# Fields the raw registry payload must supply before we will instantiate an entry.
REQUIRED_FIELDS = ("puid", "name", "version", "mimeType", "signatureOffset", "signature")
VALID_RISK = {"LOW", "MODERATE", "HIGH", "UNASSESSED"}


@dataclass(frozen=True)
class RegistryResponse:
    """Immutable, auditable normalisation of a single registry entry."""
    format_id: str
    format_name: str
    version: str
    mime_type: str
    signature_offset: int
    signature_bytes: str
    risk_level: str
    registry_version: str
    query_timestamp: str
    response_hash: str


def compute_audit_hash(payload: bytes) -> str:
    """Deterministic SHA-256 over the canonical registry payload."""
    return hashlib.sha256(payload).hexdigest()


def fetch_registry_entry(format_id: str, registry_url: str) -> Optional[Dict[str, Any]]:
    """Poll an internal registry mirror with a hard timeout and typed error handling."""
    request = urllib.request.Request(
        f"{registry_url}/api/v1/format/{format_id}",
        headers={"Accept": "application/json", "User-Agent": "OAIS-Preservation-Engine/1.0"},
    )
    try:
        with urllib.request.urlopen(request, timeout=15) as response:
            return json.loads(response.read().decode("utf-8"))
    except (urllib.error.URLError, urllib.error.HTTPError, json.JSONDecodeError) as exc:
        logger.error("Registry query failed for %s: %s", format_id, exc)
        return None


def resolve_format(format_id: str, registry_base: str, registry_version: str) -> Optional[RegistryResponse]:
    """Fetch, canonicalise, hash, validate, and normalise a registry entry."""
    logger.info("Resolving format %s against registry %s", format_id, registry_version)
    raw = fetch_registry_entry(format_id, registry_base)
    if raw is None:
        return None

    # Canonical serialisation makes the audit hash stable regardless of key order.
    canonical = json.dumps(raw, sort_keys=True, separators=(",", ":")).encode("utf-8")
    response_hash = compute_audit_hash(canonical)

    missing = [field for field in REQUIRED_FIELDS if field not in raw]
    if missing:
        logger.warning("Registry entry %s missing required fields: %s", format_id, missing)
        return None

    risk = str(raw.get("sustainability", "UNASSESSED")).upper()
    if risk not in VALID_RISK:
        logger.warning("Unknown risk level %r for %s; defaulting to UNASSESSED", risk, format_id)
        risk = "UNASSESSED"

    logger.info("Resolved %s | audit_hash=%s", format_id, response_hash[:12])
    return RegistryResponse(
        format_id=raw["puid"],
        format_name=raw["name"],
        version=raw["version"],
        mime_type=raw["mimeType"],
        signature_offset=int(raw["signatureOffset"]),
        signature_bytes=raw["signature"],
        risk_level=risk,
        registry_version=registry_version,
        query_timestamp=datetime.now(timezone.utc).isoformat(),
        response_hash=response_hash,
    )


def map_to_premis(entry: RegistryResponse) -> Dict[str, Any]:
    """Transform a validated registry entry into PREMIS-compliant technical metadata."""
    return {
        "objectCategory": "representation",
        "formatDesignation": {
            "formatName": entry.format_name,
            "formatVersion": entry.version,
        },
        "formatRegistry": {
            "formatRegistryName": "PRONOM",
            "formatRegistryKey": entry.format_id,
            "formatRegistryRole": f"specification (release {entry.registry_version})",
        },
        "formatNote": entry.mime_type,
        "preservationLevel": "full" if entry.risk_level in {"LOW", "MODERATE"} else "migrate",
        "linkingEventIdentifier": {
            "eventType": "format identification",
            "eventDateTime": entry.query_timestamp,
            "eventDetail": "Automated registry resolution via OAIS-compliant pipeline",
            "eventOutcome": entry.response_hash,
        },
    }


if __name__ == "__main__":
    # PRONOM does not expose a public REST API; mirror the signature files locally
    # or resolve against a private registry clone in production.
    TARGET_REGISTRY = "https://registry.example.internal"
    SIGNATURE_RELEASE = "V120"
    FORMAT_PUID = "fmt/43"  # JPEG (raw JFIF)

    resolved = resolve_format(FORMAT_PUID, TARGET_REGISTRY, SIGNATURE_RELEASE)
    if resolved is None:
        logger.error("Resolution pipeline halted: registry entry unavailable or invalid.")
    else:
        print(json.dumps(map_to_premis(resolved), indent=2))

The canonical serialisation before hashing is deliberate: sorting keys and stripping insignificant whitespace makes the SHA-256 digest stable regardless of how the registry orders its JSON, so the same logical entry always yields the same audit hash. The preservationLevel derivation is where the risk score earns its place — a HIGH-risk format is stamped migrate at resolution time, seeding the normalisation queue before the object ever reaches archival storage.

# Integration Points

Format registry integration is a mid-pipeline stage, so most of its value comes from the contracts it honours at each edge. Its input is the raw PUID produced by Preservation Format Identification; a file that identification cannot resolve is quarantined here rather than passed on with a guessed identity. Its primary output is the PREMIS format designation consumed by PREMIS Metadata Mapping, which folds the formatRegistry block and the linked identification event into the object’s provenance record.

Upstream in the capture pipeline, the technical characteristics a device reports are reconciled against registry truth by the ingestion-side Metadata Extraction Workflows stage, so a scanner that mislabels a JPEG 2000 master as a plain TIFF is caught before archival commitment. The atomic signature-database swaps this stage performs are replicated by Long-Term Storage Architecture, and the whole subsystem operates under the enforcement points defined by Digital Preservation Security Policies.

# Validation and Compliance Rules

Registry outputs must be normalised into institutional metadata before ingestion, and that normalisation carries hard compliance obligations. Each resolution emits exactly one PREMIS event of type format identification, cross-referenced to the signature-file release (the agent), the resolved PUID, and the deterministic response hash (the outcome). Under ISO 16363, the trusted repository must demonstrate that it monitors format obsolescence (clause 4.2.3) and that every preservation action is traceable (clause 4.1.8) — the registry-version pin and the audit hash are precisely the evidence an auditor probes for.

The mapping layer operates under strict Digital Preservation Security Policies: role-based access to the signature-update pipeline, immutable audit trails, and cryptographic verification of every metadata payload. All registry interactions, PREMIS mappings, and hashes should be persisted in a write-once ledger to satisfy compliance review. As cryptographic standards evolve, institutions should begin evaluating post-quantum hash schemes to future-proof the integrity verification of registry-derived metadata and format signatures against next-generation computational threats.

# Troubleshooting Reference

# Frequently Asked Questions

# Why pin the registry version when resolving a format?

Because PRONOM periodically ships signature files that reassign a PUID for a legacy format. An unpinned pipeline silently rewrites its own historical identification baseline when that happens, so two files ingested a year apart can end up with different recorded identities for the same bytes. Recording the exact signature-file release each entry was resolved against makes every identification reproducible and turns registry drift into a visible, auditable diff instead of a hidden corruption.

# Should the ingest path call PRONOM directly?

No. The public registry is a synchronisation source, never a request-path dependency. Resolving against it directly couples ingest availability to an external service and collapses under load during a capture surge. Mirror the signature files internally, resolve every lookup against that mirror, and update the mirror through a gated, atomic pipeline that regression-tests each delta before promotion.

# How is a format identity tied to an auditable chain of custody?

The raw registry payload is serialised to canonical JSON — sorted keys, no insignificant whitespace — and hashed with SHA-256, so the same logical entry always yields the same digest. That hash is recorded as the outcome of a PREMIS format-identification event alongside the PUID and the signature-file release, and the event is written to append-only storage. An auditor can later re-hash the recorded payload and confirm it matches the identity stamped on the archival object.

# What happens when a format crosses the risk threshold?

The composite risk score is evaluated at resolution time. When it crosses the migration threshold, the object’s preservationLevel is stamped migrate rather than full, which seeds the normalisation queue. This is what makes obsolescence handling proactive: a dying format is flagged the moment it enters the archive, feeding Preservation Planning instead of waiting for a rendering failure to expose it years later.

# Which registries should feed the integration layer?

PRONOM is the primary source for signatures, PUIDs, and byte offsets. The Library of Congress Sustainability of Digital Formats resource supplies the narrative sustainability factors that drive the risk model, and Wikidata can supplement adoption signals. All three are normalised into the single internal registry-entry contract before any of their values reaches a preservation decision, so a change in one upstream schema never propagates into the pipeline.

# Related

• Preservation Format Identification — the signature-matching stage that supplies the PUID this subsystem resolves.
• Configuring format identification tools like DROID — extending PRONOM signature coverage for rare and proprietary formats.
• PREMIS Metadata Mapping — folds the registry format designation and identification event into object provenance.
• Long-Term Storage Architecture — replicates the atomic signature-database swaps across archival nodes.
• Digital Preservation Security Policies — the access controls and immutable-audit rules the update pipeline runs under.
• Metadata Extraction Workflows — reconciles device-reported technical characteristics against registry truth at capture.