Versioning JSON Schema Profiles for Ingest Batches

An ingest batch is only ever conformant relative to the exact validation profile it was checked against. The moment that profile is edited in place — a new required field, a tightened enum, a stricter pattern — every batch validated before the edit becomes ambiguous: was it genuinely compliant, or merely compliant against rules that no longer exist? This page, part of the Batch Validation Schemas area, is the reference for versioning the JSON Schema profiles that ingest batches are validated against, so that a profile change never silently invalidates a batch that already passed, nor silently passes an old batch under rules it was never meant to satisfy. Where validating schema compliance during digital ingest covers whether a manifest conforms, this page covers which profile it must conform to, and how that binding survives years of producer and policy change.

# Root-Cause Analysis of Profile Churn

Validation profiles are not static. They evolve under three independent pressures, and each one, applied to a mutable profile, corrupts the reproducibility of prior validations.

1. Producer tooling changes. A scanner vendor ships new firmware that renames colorProfile to iccProfileName, or a capture station begins emitting a captureSoftwareVersion field the profile never anticipated. The manifest shape drifts out from under the profile — the same drift that detecting schema drift across scanner firmware versions diagnoses at the fleet level. If the profile is patched to accommodate the new shape without a version boundary, batches from the old firmware and the new firmware are now judged by a blended rule set that matches neither.

2. New required fields. Preservation policy tightens: checksumAlgorithm becomes mandatory so that every object carries an audit-grade SHA-256 declaration; a rightsStatementUri is now required for restricted collections. Adding a field to required is a breaking change — every batch that validated cleanly yesterday fails today, not because the batch is defective but because the goalposts moved. Re-running last year’s accession against today’s profile produces a wall of false failures.

3. Tightened enums and patterns. An enum of accepted formatPuid values is narrowed to drop a deprecated codec; a resolutionDpi constraint gains a minimum; a loose string becomes a pattern-constrained identifier. Each tightening retroactively criminalizes submissions that were legitimately accepted under the looser rule.

The common failure is treating the profile as a live document rather than a series of immutable, addressable artifacts. When the profile at schemas/batch.json is simply overwritten, there is no way to answer the auditor’s question — “under exactly which rules did this 2024 batch pass?” — because those rules have been destroyed. The resolution is to version profiles semantically, give every version an immutable $id, pin the profile version into each batch manifest, and load the exact pinned profile at validation time. The diagram below traces that resolution path.

The resolver binds each batch to the exact profile version its manifest pins; an unknown or missing version fails closed rather than silently defaulting to the latest.

# Semantic Versioning of Validation Profiles

Adopt semantic versioning for profiles, but define the axes in terms of what a version bump does to previously valid batches, not in terms of source-code compatibility. The distinction that matters for an archive is: does this change make a formerly valid batch invalid (breaking), or does it only accept more than before (additive)?

• MAJOR — a breaking change. Any batch that was valid under N.x.y may now be invalid under (N+1).0.0. Adding a required field, tightening an enum, adding a pattern, or lowering a maximum all qualify. A major bump demands a migration decision for historical batches.
• MINOR — an additive, backward-compatible change. Everything valid under the old version is still valid: adding an optional property, widening an enum, relaxing a bound, or adding a new permitted format. No historical batch is retroactively invalidated.
• PATCH — clarification with no constraint change: a corrected description, an added $comment, tightened metadata that does not alter which documents validate.

The table below maps concrete profile edits to the required version bump and the migration action for batches already recorded under an earlier profile.

The policy that keeps this coherent is simple and non-negotiable: additive changes may ship as a minor bump; anything that could invalidate a previously valid batch is a major bump and requires a new immutable $id. Never edit a published profile in place. A version, once emitted, is frozen forever, because batches out in storage point at it by identity.

# Immutable $id URIs Per Version

Every profile version carries a version-scoped, immutable $id. The $id is the profile’s permanent address; it is what a manifest pins and what the resolver dereferences. Because it encodes the version, 2.0.0 and 2.1.0 are distinct documents that coexist indefinitely.

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$id": "https://schemas.archival-digitization.org/ingest/batch/2.1.0.json",
  "title": "Ingest Batch Manifest Profile",
  "type": "object",
  "required": ["batchId", "profileVersion", "checksumAlgorithm", "objects"],
  "properties": {
    "batchId": {"type": "string", "pattern": "^[A-Z0-9-]{6,}$"},
    "profileVersion": {"type": "string", "const": "2.1.0"},
    "checksumAlgorithm": {"type": "string", "enum": ["SHA-256", "SHA-512"]},
    "objects": {
      "type": "array",
      "minItems": 1,
      "items": {
        "type": "object",
        "required": ["filename", "sha256", "formatPuid"],
        "properties": {
          "filename": {"type": "string"},
          "sha256": {"type": "string", "pattern": "^[a-f0-9]{64}$"},
          "formatPuid": {"type": "string", "pattern": "^fmt/[0-9]+$"}
        }
      }
    }
  }
}

Pinning profileVersion with const inside the profile means a manifest cannot claim to be 2.1.0 while being validated by 2.0.0 — the version declaration and the resolved document must agree, which turns a mis-pin into an immediate validation failure rather than a silent mismatch.

# The Versioned Profile Registry and Resolver

The resolver is the load-bearing component. It reads the profileVersion a batch pins, loads the exact profile registered under that version, validates against it, and records the resolved $id so the provenance record names the precise rules applied. Crucially, it fails closed: an unknown or absent version is never quietly upgraded to the newest profile.

from __future__ import annotations

import json
import logging
from dataclasses import dataclass
from pathlib import Path
from typing import Any

from jsonschema import Draft202012Validator
from jsonschema.exceptions import ValidationError

logger = logging.getLogger("archival.ingest.profiles")


class UnknownProfileVersion(Exception):
    """Raised when a batch pins a profile version absent from the registry."""


@dataclass(frozen=True)
class Profile:
    """An immutable, version-addressed validation profile."""

    version: str
    schema_id: str
    schema: dict[str, Any]


class ProfileRegistry:
    """Loads frozen JSON Schema profiles keyed by semantic version.

    Profiles are read once from a directory of immutable files named
    ``<major>.<minor>.<patch>.json`` and never mutated in place.
    """

    def __init__(self, profile_dir: Path) -> None:
        self._profiles: dict[str, Profile] = {}
        for path in sorted(profile_dir.glob("*.json")):
            schema = json.loads(path.read_text(encoding="utf-8"))
            version = str(schema["properties"]["profileVersion"]["const"])
            profile = Profile(version=version, schema_id=str(schema["$id"]), schema=schema)
            # Guard against two files claiming the same version.
            if version in self._profiles:
                raise ValueError(f"Duplicate profile version registered: {version}")
            # Validate the schema document itself against its declared meta-schema.
            Draft202012Validator.check_schema(schema)
            self._profiles[version] = profile
            logger.info(
                "profile_registered",
                extra={"version": version, "schema_id": profile.schema_id},
            )

    def resolve(self, version: str) -> Profile:
        """Return the exact profile pinned, or fail closed if unregistered."""
        profile = self._profiles.get(version)
        if profile is None:
            logger.error(
                "profile_version_unknown",
                extra={"requested_version": version,
                       "registered_versions": sorted(self._profiles)},
            )
            raise UnknownProfileVersion(version)
        return profile


def validate_batch(manifest: dict[str, Any], registry: ProfileRegistry) -> dict[str, Any]:
    """Validate a batch against the profile version it pins in its manifest.

    Returns a PREMIS-style validation event that records the resolved profile
    $id so provenance names the exact rules applied. Fails closed when the
    manifest omits a version or pins one the registry does not know.
    """
    batch_id = str(manifest.get("batchId", "UNKNOWN"))
    pinned = manifest.get("profileVersion")
    if not isinstance(pinned, str):
        logger.error("profile_version_missing", extra={"batch_id": batch_id})
        raise UnknownProfileVersion("<missing>")

    profile = registry.resolve(pinned)
    validator = Draft202012Validator(profile.schema)
    errors = sorted(validator.iter_errors(manifest), key=lambda e: list(e.absolute_path))

    outcome = "valid" if not errors else "invalid"
    event = {
        "eventType": "validation",
        "batchId": batch_id,
        "profileVersion": profile.version,
        "profileSchemaId": profile.schema_id,
        "outcome": outcome,
        "detail": [_format_error(e) for e in errors],
    }
    logger.info(
        "batch_validated",
        extra={"batch_id": batch_id, "outcome": outcome,
               "profile_schema_id": profile.schema_id, "error_count": len(errors)},
    )
    return event


def _format_error(error: ValidationError) -> str:
    """Render one validation error as a stable, auditable string."""
    location = "/".join(str(p) for p in error.absolute_path) or "<root>"
    return f"{location}: {error.message}"

The resolver never guesses. If a manifest pins 2.1.0, it is validated against 2.1.0 or it is not validated at all — there is no implicit “use the latest” path that would let a change to the newest profile retroactively re-judge old work. This is the same discipline the pipeline applies when batches flow through async task queuing for batches: the profile version travels with the payload so any worker, on any node, at any later date, validates against identical rules.

# Validation and Verification

Confirm the versioning scheme actually delivers reproducibility rather than merely looking correct:

• Reproduce a historical validation. Re-run a 2024 batch through the resolver. Because it pins 1.0.0 and that profile is frozen in the registry, the recorded profileSchemaId in the new event must be byte-identical to the one in the original event. If it names a newer $id, the resolver is silently upgrading — a defect.
• Assert fail-closed behaviour. Feed the resolver a manifest pinning a version that does not exist. It must raise UnknownProfileVersion and route the batch to quarantine, never fall through to the newest profile.
• Assert the const guard. Take a 2.0.0-shaped manifest, relabel its profileVersion to 2.1.0, and validate. The const on profileVersion (plus any newly required fields) must reject it, proving a mis-pin cannot masquerade as conformance.
• Diff two versions before publishing. Load the candidate profile and its predecessor and confirm the semver bump matches the change class in the table above. A new entry in required accompanied by only a minor bump is a release blocker — validate the meta-schema and the bump together in CI.

Every validation writes a validation event naming the resolved $id; that event, folded into object provenance, is what lets an auditor answer which rules applied when years later.

# Edge Cases and Gotchas

• $ref targets must be versioned too. If batch/2.1.0.json references a shared object-core.json via $ref, and that shared document is edited in place, you have reintroduced mutability through the back door. Version referenced sub-schemas the same way and pin them by immutable $id, or the “frozen” parent silently changes meaning.
• Grandfathering vs. remediation. A major bump forces a choice per historical batch: keep it validatable under its original pinned $id (grandfather), or transform its manifest to the new shape and re-validate. Both are legitimate; what is not legitimate is deleting the old profile, which makes the grandfathered batches unverifiable.
• Draft dialect drift. The $schema meta-schema (Draft 2020-12 above) is itself a version. Migrating profiles from Draft-07 to 2020-12 can change how items, additionalProperties, or $ref resolve. Treat a dialect migration as a major bump and re-validate a representative sample, because identical-looking keywords can validate differently across drafts.
• Latin-1 and byte-level manifest quirks. A manifest emitted by a legacy capture station in Latin-1 rather than UTF-8 can validate structurally while carrying mojibake in string fields. Normalize encoding before validation; JSON Schema checks structure, not byte provenance.
• Registry drift across nodes. If two workers hold different snapshots of the profile directory, the same batch can validate differently depending on where it lands. Ship the registry as an immutable, checksummed artifact so every node resolves 2.1.0 to the identical bytes.

# Frequently Asked Questions

# When is a profile change a minor bump versus a major bump?

Ask a single question: could this change make a batch that was valid yesterday invalid today? If yes — a new required field, a tightened enum, an added pattern, a lowered maximum, a removed property — it is a MAJOR bump and needs a new immutable $id. If the change only ever accepts more than before — an optional property, a widened enum, a relaxed bound — it is backward-compatible and ships as a MINOR bump. Pure clarifications that change no constraint are PATCH. The axis is the effect on previously valid batches, not the size of the diff.

# Why pin the profile version in the manifest instead of always using the latest?

Because “the latest” is a moving target, and validating a historical batch against it destroys reproducibility. A batch captured under 1.0.0 was accepted under 1.0.0’s rules; re-judging it against 3.0.0 answers a question nobody asked and manufactures false failures the moment policy tightens. Pinning the version into the manifest binds the batch to the exact rules in force at ingest, so any later re-validation reproduces the original verdict rather than reflecting subsequent policy changes.

# What should the resolver do when a manifest pins an unknown profile version?

Fail closed. An unregistered or missing profileVersion must raise an error and route the batch to quarantine — never fall back to the newest profile. A silent upgrade would validate the batch against rules it was never meant to satisfy, either passing it under looser constraints or failing it under stricter ones, and in both cases the recorded provenance would name the wrong ruleset. Refusing to validate is the safe outcome; it surfaces the mis-pin for a human to resolve instead of burying it.

# How do I handle batches already ingested when a breaking change ships?

You have two defensible options and one forbidden one. You may grandfather them: keep the old profile frozen in the registry under its original $id so those batches remain reproducibly validatable under the rules they were accepted against. Or you may remediate: transform each manifest into the new shape and re-validate against the new profile, recording a migration event. What you may never do is delete or overwrite the old profile, because that leaves every batch pinned to it permanently unverifiable.

# Related

• Batch Validation Schemas — the parent area: the manifest contract, JSON Schema profiles, and how validation gates ingest.
• Validating Schema Compliance During Digital Ingest — how a single manifest is checked for conformance once its profile is resolved.
• Detecting Schema Drift Across Scanner Firmware Versions — spotting the producer-side manifest changes that force a profile bump in the first place.