Promoting a Validated SIP to an AIP, Step by Step

Promotion is the moment a Submission Information Package stops being a candidate and becomes the archival record of last resort, and it is precisely where most pipelines quietly cut corners. This walkthrough belongs to the End-to-End SIP-to-AIP Promotion stage, and it gives the exact ordered sequence for turning a validated SIP into a sealed Archival Information Package without leaving a half-written package, an out-of-order event stream, or a “sealed” flag on bytes nobody has read back. The failure this page prevents is the promotion that looks finished — a row in a database says sealed — while the AIP on storage was never re-verified, its PREMIS chain skips the ingestion event, or a crash between “write” and “commit” left a torso of an object that later fixity sweeps will flag as corruption that never actually happened.

The governing discipline is the same four beats at every step: record intent, act, verify, then commit. Nothing downstream is allowed to observe a state the pipeline has not yet proven. An AIP is sealed last, only after every byte is on archival storage, re-read, and matched against a manifest, and only after the ordered event chain that an auditor will reconstruct has been emitted in the order the actions actually happened.

Where Promotion Goes Wrong

Four concrete anti-patterns account for nearly every corrupt or unauditable AIP produced at this stage:

  1. Sealing before all checks pass. The pipeline treats “SIP validation succeeded earlier” as permission to seal now. But validation and promotion are separated in time; a bit could have flipped in staging storage, or a file could have been swapped by a rerun. Promotion must re-derive fixity from the bytes it is about to archive, not trust an upstream result. This is the same re-check the batch validation schemas enforce at ingest, applied again at the storage boundary.
  2. Non-atomic seal. The AIP is written file-by-file into its final archival location, and the “sealed” marker is set partway through — or a crash leaves objects half-copied with no marker at all. A reader or a fixity sweep then encounters an AIP that is neither absent nor complete. Atomicity means the AIP becomes visible as one indivisible commit, never as a growing pile.
  3. Events emitted out of order (or not at all). PREMIS events are the provenance spine. If ingestion is recorded before the AIP is actually on storage, or replication is emitted before ingestion, the chain an auditor reconstructs describes a history that never happened. Events must be emitted after the action they describe verifiably completed, in causal order.
  4. No verify-on-read. The pipeline computes a digest as it writes, then trusts the write. A silent storage fault — a truncated object, a dropped write on a full volume — is invisible until years later. Promotion must read the archived bytes back and re-hash them before it will seal.

The state diagram below shows the whole promotion as an ordered sequence of gated transitions. Every stage either advances to the next gate or diverts to a fail-closed terminal state; nothing skips ahead, and SEALED is reachable only through the verify-on-read gate.

Ordered, fail-closed SIP-to-AIP promotion sequence A left-to-right ordered sequence of eight gated stages, each a rounded box. Stage 1 Re-verify fixity, stage 2 Identify formats, stage 3 Assemble PREMIS and METS, stage 4 Compute AIP manifest and digests, stage 5 Write to archival storage atomically, stage 6 Verify on read, stage 7 Emit ingestion then replication events, stage 8 Mark AIP sealed. Solid teal arrows connect each stage to the next in order. From each of the first six gates a dashed amber arrow drops to a single fail-closed terminal state at the bottom labelled Abort and emit failure event, never sealed. The final transition into SEALED is only reachable after the Verify on read gate passes, showing that promotion commits last. STEP 1 Re-verify fixity SHA-256 from bytes STEP 2 Identify formats PRONOM PUID / every file STEP 3 Assemble metadata PREMIS + METS STEP 4 Compute manifest AIP-level digests STEP 5 Write atomically staging → final commit STEP 6 Verify on read re-hash archived bytes STEP 7 Emit events ingestion → replication STEP 8 Mark AIP SEALED atomic commit · last FAIL-CLOSED TERMINAL Abort · emit failure event nothing sealed · SIP left intact any gate fails →

Promotion as an ordered, gated sequence: each step must pass before the next begins, every gate can divert to one fail-closed terminal, and SEALED is committed last — only after the archived bytes are re-read and verified.

Step-by-Step Promotion

The steps below follow the diagram in order. Each corresponds to a helper that raises on failure; the orchestrating promote_sip_to_aip() at the end wires them into a single fail-closed transaction. The scaffolding types and logger come first.

python
from __future__ import annotations

import hashlib
import json
import logging
import os
import shutil
import tempfile
from dataclasses import dataclass, field
from datetime import datetime, timezone
from pathlib import Path

logger = logging.getLogger("archival.promotion")

READ_CHUNK = 1 << 20  # 1 MiB streaming reads keep large masters off the heap


class PromotionError(RuntimeError):
    """Raised on any failed promotion gate; the pipeline fails closed."""


@dataclass(frozen=True)
class SipFile:
    """One file inside the SIP, with the fixity recorded at validation time."""

    relpath: str
    source_path: Path
    expected_sha256: str


@dataclass
class PromotionContext:
    """Carries state across the ordered promotion steps."""

    sip_id: str
    aip_id: str
    files: list[SipFile]
    archival_root: Path
    formats: dict[str, str] = field(default_factory=dict)   # relpath -> PRONOM PUID
    manifest_digest: str = ""

Step 1 — Confirm SIP validation and re-verify fixity

Do not trust the upstream “valid” flag. Re-derive the SHA-256 of every file from the bytes on disk and compare it to the digest recorded at validation. A mismatch here means the SIP changed between validation and promotion, and the only safe action is to fail closed before anything is written to archival storage.

python
def sha256_of(path: Path) -> str:
    """Stream a file through SHA-256 without loading it whole into memory."""
    digest = hashlib.sha256()
    with path.open("rb") as handle:
        for chunk in iter(lambda: handle.read(READ_CHUNK), b""):
            digest.update(chunk)
    return digest.hexdigest()


def reverify_fixity(ctx: PromotionContext) -> None:
    """STEP 1 — re-derive fixity from the bytes we are about to archive."""
    for entry in ctx.files:
        actual = sha256_of(entry.source_path)
        if actual != entry.expected_sha256:
            logger.error(
                "fixity_mismatch",
                extra={"sip_id": ctx.sip_id, "file": entry.relpath,
                       "expected": entry.expected_sha256, "actual": actual},
            )
            raise PromotionError(f"Fixity mismatch on {entry.relpath}")
    logger.info("fixity_reverified",
                extra={"sip_id": ctx.sip_id, "file_count": len(ctx.files)})

$$ \text{promote} ;\Longleftarrow; \bigwedge_{f \in \text{SIP}} \big(\operatorname{SHA256}(f_{\text{bytes}}) = h_f^{\text{recorded}}\big) $$

Promotion is permitted only when the conjunction holds for every file — one mismatched digest vetoes the whole package.

Step 2 — Format-identify every file

Every file gets a definitive PRONOM identification (a PUID), not a guess from its extension. This is the discipline covered in depth by the OAIS reference model implementation: an AIP without representation information is not preservable. In production this call shells out to Siegfried or DROID; here it is a typed seam you can back with either.

python
def identify_formats(ctx: PromotionContext) -> None:
    """STEP 2 — resolve a PRONOM PUID for every file; unknowns fail closed."""
    for entry in ctx.files:
        puid = resolve_puid(entry.source_path)
        if not puid:
            logger.error("format_unidentified",
                         extra={"sip_id": ctx.sip_id, "file": entry.relpath})
            raise PromotionError(f"No PRONOM identification for {entry.relpath}")
        ctx.formats[entry.relpath] = puid
    logger.info("formats_identified",
                extra={"sip_id": ctx.sip_id, "distinct_puids": len(set(ctx.formats.values()))})


def resolve_puid(path: Path) -> str:
    """Return a PRONOM PUID for a file (seam over Siegfried/DROID)."""
    # A real implementation invokes `sf -json` or the DROID API and parses the PUID.
    suffix_map = {".tif": "fmt/353", ".tiff": "fmt/353",
                  ".jp2": "x-fmt/392", ".pdf": "fmt/477"}
    return suffix_map.get(path.suffix.lower(), "")

Step 3 — Assemble metadata (PREMIS / METS)

With formats known, build the descriptive and preservation metadata that will travel inside the AIP. The PREMIS objects record fixity and format; the METS document is the structural map. Crucially, this step only assembles metadata — it does not yet emit provenance events, because the actions those events describe have not happened.

python
def assemble_metadata(ctx: PromotionContext) -> dict:
    """STEP 3 — build in-package PREMIS objects and a METS structural map."""
    premis_objects = [
        {
            "objectIdentifier": {"type": "AIP-relative", "value": e.relpath},
            "objectCharacteristics": {
                "fixity": {"messageDigestAlgorithm": "SHA-256",
                           "messageDigest": e.expected_sha256},
                "format": {"formatRegistry": "PRONOM",
                           "formatRegistryKey": ctx.formats[e.relpath]},
            },
        }
        for e in ctx.files
    ]
    mets = {
        "aip_id": ctx.aip_id,
        "created": datetime.now(timezone.utc).isoformat(),
        "fileSec": [{"path": e.relpath, "puid": ctx.formats[e.relpath]} for e in ctx.files],
    }
    logger.info("metadata_assembled",
                extra={"sip_id": ctx.sip_id, "premis_object_count": len(premis_objects)})
    return {"premis": premis_objects, "mets": mets}

Step 4 — Compute the AIP-level manifest and digests

Individual file fixity is not enough; the AIP needs one manifest that binds every path to its digest, plus a digest of the manifest itself. That top-level digest is the single value a future fixity sweep checks to prove the whole package is intact. The manifest is written into the package in Step 5 and re-verified in Step 6.

python
def compute_manifest(ctx: PromotionContext, metadata: dict) -> dict:
    """STEP 4 — build the AIP manifest and a digest that seals the whole package."""
    entries = {e.relpath: e.expected_sha256 for e in ctx.files}
    manifest = {
        "aip_id": ctx.aip_id,
        "algorithm": "SHA-256",
        "files": dict(sorted(entries.items())),   # sorted → deterministic serialization
        "metadata_present": sorted(metadata.keys()),
    }
    serialized = json.dumps(manifest, sort_keys=True, separators=(",", ":")).encode("utf-8")
    ctx.manifest_digest = hashlib.sha256(serialized).hexdigest()
    logger.info("manifest_computed",
                extra={"sip_id": ctx.sip_id, "aip_id": ctx.aip_id,
                       "manifest_digest": ctx.manifest_digest})
    return {"manifest": manifest, "serialized": serialized}

Step 5 — Write to archival storage atomically

Never build the AIP in place at its final path. Stage the whole package in a temporary directory on the same filesystem as the archival root, then make it visible with a single atomic os.replace. If the process dies mid-copy, the final location still does not exist — there is no torso to clean up, and no reader ever sees a partial AIP.

python
def write_atomically(ctx: PromotionContext, metadata: dict, manifest: dict) -> Path:
    """STEP 5 — stage the AIP, then publish it with one atomic rename."""
    final_path = ctx.archival_root / ctx.aip_id
    if final_path.exists():
        raise PromotionError(f"AIP {ctx.aip_id} already exists; refusing to overwrite")

    staging = Path(tempfile.mkdtemp(dir=ctx.archival_root, prefix=f".stage-{ctx.aip_id}-"))
    try:
        (staging / "data").mkdir()
        for entry in ctx.files:
            dest = staging / "data" / entry.relpath
            dest.parent.mkdir(parents=True, exist_ok=True)
            shutil.copy2(entry.source_path, dest)
        (staging / "manifest.json").write_bytes(manifest["serialized"])
        (staging / "premis.json").write_text(json.dumps(metadata["premis"], indent=2))
        (staging / "mets.json").write_text(json.dumps(metadata["mets"], indent=2))

        # fsync directory metadata so the rename survives a power loss.
        dir_fd = os.open(staging, os.O_RDONLY)
        try:
            os.fsync(dir_fd)
        finally:
            os.close(dir_fd)

        os.replace(staging, final_path)   # atomic on a POSIX filesystem
    except Exception:
        shutil.rmtree(staging, ignore_errors=True)
        raise
    logger.info("aip_written", extra={"sip_id": ctx.sip_id, "aip_id": ctx.aip_id,
                                      "path": str(final_path)})
    return final_path

Step 6 — Verify on read

Now read the archived bytes back from their final location and re-hash them. This is the gate that catches a truncated copy or a silent storage fault. Only if the re-read manifest digest matches the value computed in Step 4 does promotion proceed toward sealing.

python
def verify_on_read(ctx: PromotionContext, aip_path: Path) -> None:
    """STEP 6 — re-read archived bytes and confirm nothing changed in transit."""
    for entry in ctx.files:
        archived = aip_path / "data" / entry.relpath
        actual = sha256_of(archived)
        if actual != entry.expected_sha256:
            raise PromotionError(f"Read-back mismatch on {entry.relpath}")

    manifest_bytes = (aip_path / "manifest.json").read_bytes()
    redigest = hashlib.sha256(manifest_bytes).hexdigest()
    if redigest != ctx.manifest_digest:
        raise PromotionError("Manifest digest changed on read-back")
    logger.info("verify_on_read_ok",
                extra={"sip_id": ctx.sip_id, "aip_id": ctx.aip_id})

Step 7 — Emit ingestion, then replication, events

Only now — with verified bytes on storage — are the provenance events true. Emit ingestion first, then replication, in that causal order; each event carries the manifest digest so an auditor can bind the event to the exact package state. The event-emission contract itself is defined by emitting PREMIS events for preservation actions; here we simply respect its ordering.

python
def emit_event(ctx: PromotionContext, event_type: str, outcome: str) -> None:
    """Append one ordered PREMIS event to the provenance ledger."""
    event = {
        "eventType": event_type,
        "eventDateTime": datetime.now(timezone.utc).isoformat(),
        "eventOutcome": outcome,
        "linkedObject": ctx.aip_id,
        "manifestDigest": ctx.manifest_digest,
    }
    record_premis_event(event)   # durable, append-only sink
    logger.info("premis_event_emitted",
                extra={"sip_id": ctx.sip_id, "aip_id": ctx.aip_id,
                       "event_type": event_type, "outcome": outcome})


def record_premis_event(event: dict) -> None:
    """Persist a PREMIS event to the append-only preservation ledger (seam)."""
    # A real sink writes to the immutable audit ledger; kept side-effect-free here.
    logger.debug("ledger_append", extra={"event": event})

Step 8 — Mark the AIP sealed

Sealing is a single atomic state transition, and it happens last. The sealed marker means “every prior gate passed and the record is immutable.” Because the events were already emitted and verified, the seal never precedes the evidence for it.

python
def mark_sealed(ctx: PromotionContext, aip_path: Path) -> None:
    """STEP 8 — commit the seal atomically as the final action."""
    marker = aip_path / "SEALED"
    tmp = aip_path / ".SEALED.tmp"
    tmp.write_text(json.dumps({"aip_id": ctx.aip_id,
                               "manifest_digest": ctx.manifest_digest,
                               "sealed_at": datetime.now(timezone.utc).isoformat()}))
    os.replace(tmp, marker)   # atomic flip to sealed
    logger.info("aip_sealed", extra={"sip_id": ctx.sip_id, "aip_id": ctx.aip_id})

The orchestrator: ordered, verified, fail-closed

promote_sip_to_aip() runs the eight steps in order. Any raised PromotionError aborts before the seal, emits a failure event, and leaves the SIP untouched — the fail-closed guarantee the diagram encodes.

python
def promote_sip_to_aip(ctx: PromotionContext) -> Path:
    """Promote a validated SIP to a sealed AIP via ordered, verified, atomic steps."""
    logger.info("promotion_started", extra={"sip_id": ctx.sip_id, "aip_id": ctx.aip_id})
    aip_path: Path | None = None
    try:
        reverify_fixity(ctx)                             # STEP 1
        identify_formats(ctx)                            # STEP 2
        metadata = assemble_metadata(ctx)                # STEP 3
        manifest = compute_manifest(ctx, metadata)       # STEP 4
        aip_path = write_atomically(ctx, metadata, manifest)  # STEP 5
        verify_on_read(ctx, aip_path)                    # STEP 6
        emit_event(ctx, "ingestion", "success")          # STEP 7 (ordered)
        emit_event(ctx, "replication", "success")
        mark_sealed(ctx, aip_path)                       # STEP 8
    except PromotionError as exc:
        emit_event(ctx, "ingestion", f"failure: {exc}")
        if aip_path and aip_path.exists() and not (aip_path / "SEALED").exists():
            shutil.rmtree(aip_path, ignore_errors=True)  # remove the unsealed torso
        logger.error("promotion_failed",
                     extra={"sip_id": ctx.sip_id, "aip_id": ctx.aip_id, "error": str(exc)})
        raise
    logger.info("promotion_completed", extra={"sip_id": ctx.sip_id, "aip_id": ctx.aip_id})
    return aip_path

Validation and Verification

A promotion that returns without raising is not proof on its own; confirm the seal with independent checks rather than trusting the return value:

  • Re-derive the top-level digest from storage. Read manifest.json back from the sealed AIP and hash it; it must equal the manifestDigest recorded on the ingestion and replication events. If they diverge, the seal is describing bytes that changed.
  • Assert event order and completeness. Query the provenance ledger for the AIP and confirm exactly one ingestion event precedes exactly one replication event, both with success outcomes, both stamped before the seal timestamp. A missing or reordered event is a promotion bug even when the bytes are fine.
  • Confirm the seal is the last write. The SEALED marker’s mtime must be later than every data file and every event. If any object is newer than the seal, something wrote after commit and the AIP is no longer trustworthy.
  • Prove atomicity by fault injection. Kill the process between Step 5 and Step 8 in a staging environment and confirm the final path either does not exist or contains no SEALED marker — never a half-package that reads as complete.

Edge Cases and Gotchas

Failure mode Root cause Consequence if ignored Correct handling
Cross-device rename fails Staging dir on a different filesystem than the archival root os.replace raises OSError (EXDEV); no atomicity Create the staging dir inside archival_root so the rename is same-device
Multi-page TIFF partial read Streaming stops early on a truncated master Read-back digest passes on a short file Compare byte length as well as digest; fail on size drift
Unidentified proprietary format Scanner extension not in PRONOM identify_formats blocks promotion Register a local PUID or normalize to TIFF/JP2 before promoting
Duplicate AIP id on rerun A retried batch reuses aip_id Silent overwrite of a sealed record write_atomically refuses when the final path already exists
Event emitted before verify ingestion logged in Step 5 Provenance claims success on unverified bytes Emit events only in Step 7, after verify-on-read
Result-store expiry Events kept only in a volatile backend Audit trail evaporates Persist events to the durable append-only ledger, not a cache

Three archival-specific traps deserve extra care. First, legacy Latin-1 filenames inside older SIPs can round-trip badly through JSON — normalize paths to UTF-8 NFC before they enter the manifest, or two “identical” filenames will produce different digests. Second, a byte-order or byte-offset error in a bespoke scanner container can leave a file whose digest is stable but whose internal structure is corrupt; format identification catches this only if the tool validates structure (JHOVE-style), not just magic bytes. Third, when a gate fails after Step 5, the orchestrator removes the unsealed torso — but never a package that already carries a SEALED marker; recovering from a failure that straddles the seal is the job of handling AIP promotion rollback on fixity failure, which treats a sealed-then-suspect AIP as an immutable record to quarantine rather than delete.

For the canonical semantics of the events emitted here, consult the Library of Congress PREMIS Data Dictionary, and for the PRONOM identifiers used in Step 2, the PRONOM registry. Both are the same references the rest of the pipeline treats as authoritative.

Frequently Asked Questions

Why re-verify fixity during promotion if the SIP was already validated?

Because validation and promotion are separated in time and storage. Between the two, a bit can flip in staging, a rerun can swap a file, or a partial write can truncate a master. Trusting the earlier “valid” flag means sealing whatever is on disk now, which may not be what was validated. Re-deriving SHA-256 from the exact bytes about to be archived closes that gap, and it costs one streaming read per file — cheap insurance against sealing corruption as though it were the record.

What makes the seal atomic, and why does it matter?

The AIP is assembled in a temporary directory on the same filesystem and made visible with a single os.replace, and the SEALED marker is flipped with a second atomic rename as the very last action. Atomicity matters because a crash at any earlier point leaves either no final AIP or an unsealed package the pipeline knows to discard — never a half-written package that a reader or a fixity sweep mistakes for a complete, trustworthy record. Without atomicity, “sealed” would be a lie told partway through a copy.

Why must ingestion and replication events be emitted in that specific order?

PREMIS events are the provenance spine an auditor replays to reconstruct what happened. Replication is only meaningful after ingestion has verifiably placed the AIP on archival storage, so emitting replication first — or emitting either before verify-on-read — records a causal history that never occurred. Emitting ingestion then replication, both after the bytes are read back and matched, keeps the ledger a faithful account rather than an optimistic one.

What happens if a step fails after the AIP is already written?

The orchestrator fails closed: it emits a failure event and removes the unsealed torso from archival storage, leaving the source SIP intact for a clean retry. It never deletes a package that already carries a SEALED marker, because a sealed AIP is an immutable record. A failure discovered after sealing is a rollback-and-quarantine problem, handled separately, not something the promotion path is allowed to silently overwrite.

More in End-to-End SIP-to-AIP Promotion

Building a SIP-to-AIP Compliance Checklist A prose checklist pinned to a wiki is not a control. It is a description of a control that a tired operator is trusted to apply by hand, and it rots the… Handling AIP Promotion Rollback on Fixity Failure Promoting a submission package into the archival tier is not a single atomic write — it is a sequence of independent side effects: bytes land on…