End-to-End SIP-to-AIP Promotion: A Single Worked Path From Validated Submission to Sealed Archive

Most preservation systems are documented as two separate worlds: an ingestion floor that scans, queues, and validates incoming material, and an archival back end that stores, replicates, and audits it. In production, the trustworthiness of the whole repository is decided in the narrow seam where those worlds meet — the moment a validated Submission Information Package (SIP) is promoted into a sealed Archival Information Package (AIP). This is the walkthrough for that seam. It sits above the full archival digitization library and deliberately stitches together the two areas that own the halves of the journey: the automated ingestion and batch scanning workflows that produce a clean SIP, and the OAIS-compliant digital preservation architecture that receives it and holds it for decades. Rather than restating either area, this page follows one package end to end and shows exactly where control passes from one to the other.

The reason the SIP→AIP boundary deserves its own walkthrough is that it is where OAIS and ISO 16363 trust is won or lost. Upstream of it, a batch is still provisional: files can be rejected, re-scanned, or discarded with no lasting consequence. The instant a SIP is promoted, the repository is making a durable claim — this object is authentic, its integrity is provable, and every action taken on it is on the record. An auditor does not test that claim by reading your storage brochure; they test it by asking you to reconstruct the exact sequence of checks, digests, and events that turned a producer’s drop folder into an immutable archival object. If any link in that chain is implicit, unlogged, or reversible without a record, the promotion is not defensible. This walkthrough makes every link explicit.

To keep the path concrete, three focused guides drill into the parts most teams get wrong: a step-by-step run through the happy path in promoting a validated SIP to AIP step by step, the auditor-facing view in building a SIP-to-AIP compliance checklist, and the failure path that matters most in handling AIP promotion rollback on fixity failure. Everything below is the map those three guides hang from.

# The Worked Example That Threads This Page

Throughout this walkthrough we follow a single real-shaped package. A regional archive has digitized a bound 1924 municipal survey: 312 pages captured as archival-master TIFFs on a production scanner, plus a producer-supplied manifest and a Dublin Core descriptive record. The batch has already cleared the scanning floor — it was routed, queued, and schema-checked upstream — so what lands at the promotion boundary is a package the ingestion side considers valid. Our job is to prove that claim independently and turn it into a sealed AIP with identifier aip-1924-survey-000312. We carry that same package through every stage below, so each stage is a concrete operation on real bytes rather than an abstract phase.

The diagram traces the whole promotion, from the SIP arriving at the boundary to the sealed, replicated AIP and the append-only event stream that makes the whole thing auditable.

One validated SIP crossing the trust boundary through five gated stages into a sealed, replicated AIP, with every gate appending a signed PREMIS event and any failure diverting to quarantine instead of promoting.

# Stage 1 — Independent Fixity Verification at the Boundary

The engineering problem at stage one is trust inheritance. The SIP arrives stamped as valid by the ingestion floor, but the promotion boundary cannot inherit that verdict on faith — a manifest written on the scanning side and a byte on the archival side can diverge for entirely mundane reasons: a truncated network copy, a controller that flipped a bit in transit, or a re-export that silently re-encoded a page. So the first act of promotion is to recompute a SHA-256 digest for every file in our 312-page package and compare it, file by file, against the digests the manifest carries. This is not the same check the ingestion side ran; it is a deliberately independent re-verification performed by a different agent at a different moment, which is precisely what makes it evidence.

The upstream contract that produced those manifest digests is enforced by batch validation schemas, so the boundary check is really a handshake: the ingestion side asserts a set of digests, and the preservation side confirms them before accepting custody. In Python this is a streamed hashlib pass over each master — chunked so a 400 MB TIFF never has to be resident in memory — with the result compared against the manifest value. A match emits a messageDigestCalculation event and the file advances; a mismatch is terminal for the whole package. There is no partial promotion. If page 207 fails fixity, the survey does not become an AIP with 311 good pages; it diverts to quarantine as a unit, which is the exact scenario the rollback-on-fixity-failure guide is built around.

# Stage 2 — Signature-Based Format Identification

Once integrity is proven, the boundary needs to know what it is about to commit to a decades-long retention promise. The engineering problem is that a producer’s file extension is a claim, not a fact. Our survey pages arrive named .tif, but promotion cannot take that on trust: a renamed JPEG, a TIFF with a proprietary scanner tag that breaks strict readers, or an outright format mismatch are all integrity signals that must surface here rather than years later when a reader cannot open the master. Every bitstream is therefore classified by its internal signature and container structure, and resolved to a canonical PRONOM identifier (PUID) — for our masters, the baseline TIFF PUID fmt/353.

This is the work owned by preservation format identification, which deploys DROID or Siegfried against a current signature file and reconciles their verdicts. At the promotion boundary the Python approach is to shell out to the identifier, parse its PUID output, and apply institutional policy: an expected, sustainable format advances; an at-risk or obsolete PUID is flagged for a preservation-planning review before sealing; and a declared-versus-detected mismatch raises a quarantine event exactly as a fixity failure does. Identification is a gate, not a label — it decides whether the package is even eligible to become an AIP.

# Stage 3 — Assembling PREMIS Metadata and Provenance

With integrity and format established, the boundary assembles the metadata that will travel inside the AIP for its entire life. The engineering problem is that an AIP is only preservable if it is self-describing: a future archivist, opening this package with none of today’s context, must be able to read what the object is, where it came from, what has been done to it, and under what rights it is held. That means serializing the four PREMIS entities — Object, Event, Agent, Rights — into the package and mapping the producer’s descriptive record into them without semantic loss.

This is the domain of PREMIS metadata mapping. Our survey shipped a Dublin Core record (title, date 1924, a rights statement), and the classic failure is a lossy field-to-field crosswalk that quietly drops identifiers and event provenance; the reference procedure for doing it correctly is how to map Dublin Core to PREMIS for archival objects. In Python the approach is to build a typed PREMIS Object per master (identifier, objectCharacteristics populated from the format-ID PUID and the SHA-256 digest already computed), attach the Agent that performed the promotion, fold in the mapped Rights, and accumulate the running list of Events. Crucially, the fixity and format events from stages one and two are not discarded and re-derived — they are carried forward and embedded here, so the AIP’s provenance record is the promotion’s own audit trail.

# Stage 4 — Packaging the AIP

Stage four is where the loose parts become one archival object. The engineering problem is layout and enclosure: an AIP is not a folder of files, it is a defined package with content data, its embedded PREMIS metadata, and a tag manifest that binds them together with a checksum of every payload file — so the package’s own integrity can be verified as a whole, forever, without reference to the original SIP. Our 312 masters, their per-file PREMIS, and the collection-level descriptive metadata are laid out under aip-1924-survey-000312 following a stable structure such as BagIt or an OAIS-native container.

The packaging contract itself belongs to the OAIS reference model implementation, which defines what a valid AIP must contain and how representation information is bound to content; the worked packaging code lives in setting up OAIS SIP/AIP/DIP workflows in Python. The Python approach at this stage is deterministic assembly: write the payload tree, serialize the PREMIS document, generate a tag manifest listing every payload path with its SHA-256, and compute a single top-level package digest. That top-level digest is the AIP’s identity — it is what gets sealed, what gets replicated, and what every future fixityCheck recomputes.

# Stage 5 — Sealing, Replication, and Long-Term Custody

Sealing is the irrevocable act. The engineering problem is immutability with redundancy: the moment the package is committed, it must become unrewritable and it must exist in more than one place, because a single immutable copy is still a single point of failure. The AIP is written to a WORM-compliant, object-lock archival tier and replicated to at least one geographically distinct site, with each replica’s digest independently re-verified on write rather than assumed from a successful upload.

This is long-term storage architecture territory — tiered, immutable, geo-redundant storage with fixity re-verification woven into every read, detailed further in best practices for cold storage tiering. The Python approach is a put-then-verify contract against the storage interface: upload the package, read it back, recompute the top-level digest, confirm the object-lock retention is set and unbreakable, then repeat against each replica target. Only when every copy independently re-verifies does the promotion succeed. This is the point at which our survey stops being a batch the archive is processing and becomes aip-1924-survey-000312, an object the archive is preserving.

# Stage 6 — Emitting the Preservation Event Chain

Running underneath all five stages is the sixth, which is not sequential but continuous: every gate emits a signed PREMIS event to an append-only ledger. The engineering problem is that an action without a record is, to an auditor, an action that did not happen. Fixity verification, format identification, metadata assembly, ingestion (the promotion itself), and each replication must each land as a discrete, timestamped, agent-attributed event that chains into a tamper-evident log.

This is the concern of preservation action logging, which specifies how those events are emitted and stored so the chain cannot be silently edited or gapped. The Python approach threads a single event-emitter through the whole orchestrator so that no stage can advance the package without also writing its evidence — the event stream and the promotion are the same code path, not a logging afterthought bolted on beside it. By the time our AIP is sealed, the ledger holds the complete, ordered story of how it came to be, which is exactly the artifact an ISO 16363 assessor asks to see.

# The Promotion Orchestrator

The code below is the spine that runs stages one through six in order for our worked package. It treats the SIP as untrusted, gates each stage, emits a PREMIS event at every step, and refuses to produce an AIP unless every gate passes. A failure at any gate raises a PromotionError, which the caller handles as a quarantine-and-rollback — never as a partial success. Structured JSON logging makes every decision auditable after the fact.

import hashlib
import json
import logging
import shutil
import subprocess
import uuid
from dataclasses import dataclass, field
from datetime import datetime, timezone
from pathlib import Path
from typing import Optional

logging.basicConfig(level=logging.INFO, format="%(message)s",
                    handlers=[logging.StreamHandler()])
logger = logging.getLogger("e2e.promotion")


def _now() -> str:
    return datetime.now(timezone.utc).isoformat()


def sha256_of(path: Path, chunk: int = 1 << 20) -> str:
    """Stream a file through SHA-256 so large masters never load fully into RAM."""
    digest = hashlib.sha256()
    with path.open("rb") as handle:
        while block := handle.read(chunk):
            digest.update(block)
    return digest.hexdigest()


@dataclass(frozen=True)
class SIPFile:
    path: Path
    expected_sha256: str
    declared_format: str


@dataclass
class SIP:
    package_id: str
    files: list[SIPFile]
    dublin_core: dict[str, str]
    agent: str


@dataclass
class PREMISEvent:
    event_type: str
    outcome: str
    detail: str
    linking_object: str
    linking_agent: str
    event_id: str = field(default_factory=lambda: str(uuid.uuid4()))
    event_date_time: str = field(default_factory=_now)

    def as_record(self) -> dict[str, str]:
        return {
            "eventIdentifier": self.event_id,
            "eventType": self.event_type,
            "eventDateTime": self.event_date_time,
            "eventOutcome": self.outcome,
            "eventOutcomeDetail": self.detail,
            "linkingObjectIdentifier": self.linking_object,
            "linkingAgentIdentifier": self.linking_agent,
        }


class PromotionError(RuntimeError):
    """Raised when any gate fails; the package must divert to quarantine."""


class AuditLedger:
    """Append-only sink; a production ledger signs and chains each record."""

    def __init__(self, ledger_path: Path) -> None:
        self._path = ledger_path
        self._path.parent.mkdir(parents=True, exist_ok=True)

    def append(self, event: PREMISEvent) -> None:
        record = event.as_record()
        with self._path.open("a", encoding="utf-8") as handle:
            handle.write(json.dumps(record) + "\n")
        logger.info(json.dumps({"ledger_append": record}))


@dataclass
class AIP:
    aip_id: str
    root: Path
    package_digest: str
    events: list[PREMISEvent]


class PromotionOrchestrator:
    """Promote one validated SIP to a sealed, replicated AIP through gated stages."""

    def __init__(self, ledger: AuditLedger, staging_root: Path,
                 archival_root: Path, replica_roots: list[Path],
                 allowed_puids: frozenset[str]) -> None:
        self._ledger = ledger
        self._staging_root = staging_root
        self._archival_root = archival_root
        self._replica_roots = replica_roots
        self._allowed_puids = allowed_puids

    def _emit(self, event: PREMISEvent) -> PREMISEvent:
        self._ledger.append(event)
        return event

    def promote(self, sip: SIP) -> AIP:
        logger.info(json.dumps({"stage": "start", "package_id": sip.package_id}))
        events: list[PREMISEvent] = []
        events += self._verify_fixity(sip)
        events += self._identify_formats(sip)
        premis_doc, meta_event = self._assemble_metadata(sip, events)
        events.append(meta_event)
        aip = self._assemble_aip(sip, premis_doc, events)
        events += self._seal_and_replicate(aip)
        aip.events = events
        logger.info(json.dumps({"stage": "complete", "aip_id": aip.aip_id,
                                "event_count": len(events)}))
        return aip

    def _verify_fixity(self, sip: SIP) -> list[PREMISEvent]:
        events: list[PREMISEvent] = []
        for item in sip.files:
            observed = sha256_of(item.path)
            if observed != item.expected_sha256:
                self._emit(PREMISEvent(
                    event_type="fixityCheck", outcome="fail",
                    detail=f"expected={item.expected_sha256} observed={observed}",
                    linking_object=sip.package_id, linking_agent=sip.agent))
                raise PromotionError(f"fixity mismatch on {item.path.name}")
            events.append(self._emit(PREMISEvent(
                event_type="messageDigestCalculation", outcome="success",
                detail=f"sha256={observed} file={item.path.name}",
                linking_object=sip.package_id, linking_agent=sip.agent)))
        return events

    def _identify_formats(self, sip: SIP) -> list[PREMISEvent]:
        events: list[PREMISEvent] = []
        for item in sip.files:
            puid = self._run_identifier(item.path)
            if puid not in self._allowed_puids:
                self._emit(PREMISEvent(
                    event_type="validation", outcome="fail",
                    detail=f"puid={puid} not in allowed set for {item.path.name}",
                    linking_object=sip.package_id, linking_agent=sip.agent))
                raise PromotionError(f"format {puid} rejected for {item.path.name}")
            events.append(self._emit(PREMISEvent(
                event_type="validation", outcome="success",
                detail=f"identified {item.path.name} as {puid}",
                linking_object=sip.package_id, linking_agent=sip.agent)))
        return events

    def _run_identifier(self, path: Path) -> str:
        """Resolve a canonical PUID via Siegfried; fall back on its exit code."""
        try:
            result = subprocess.run(
                ["sf", "-json", str(path)],
                capture_output=True, text=True, timeout=120, check=True)
            payload = json.loads(result.stdout)
            matches = payload["files"][0]["matches"]
            return matches[0]["id"] if matches else "UNKNOWN"
        except (subprocess.SubprocessError, KeyError, IndexError,
                json.JSONDecodeError) as exc:
            logger.error(json.dumps({"format_id_error": str(exc),
                                     "file": path.name}))
            return "UNKNOWN"

    def _assemble_metadata(self, sip: SIP,
                           prior: list[PREMISEvent]) -> tuple[dict, PREMISEvent]:
        premis_doc = {
            "objects": [{"objectIdentifier": item.path.name,
                         "declaredFormat": item.declared_format}
                        for item in sip.files],
            "descriptive": sip.dublin_core,
            "events": [event.as_record() for event in prior],
            "agent": sip.agent,
        }
        event = self._emit(PREMISEvent(
            event_type="metadataModification", outcome="success",
            detail=f"assembled PREMIS with {len(prior)} carried events",
            linking_object=sip.package_id, linking_agent=sip.agent))
        return premis_doc, event

    def _assemble_aip(self, sip: SIP, premis_doc: dict,
                      prior: list[PREMISEvent]) -> AIP:
        aip_id = f"aip-{sip.package_id}"
        root = self._staging_root / aip_id
        payload = root / "data"
        payload.mkdir(parents=True, exist_ok=True)
        for item in sip.files:
            shutil.copy2(item.path, payload / item.path.name)
        (root / "premis.json").write_text(
            json.dumps(premis_doc, indent=2), encoding="utf-8")
        tagmanifest = {
            (p.relative_to(root)).as_posix(): sha256_of(p)
            for p in sorted(root.rglob("*")) if p.is_file()
        }
        (root / "tagmanifest-sha256.json").write_text(
            json.dumps(tagmanifest, indent=2), encoding="utf-8")
        package_digest = hashlib.sha256(
            json.dumps(tagmanifest, sort_keys=True).encode("utf-8")).hexdigest()
        self._emit(PREMISEvent(
            event_type="creation", outcome="success",
            detail=f"packaged {len(sip.files)} files; digest={package_digest}",
            linking_object=aip_id, linking_agent=sip.agent))
        return AIP(aip_id=aip_id, root=root,
                   package_digest=package_digest, events=list(prior))

    def _seal_and_replicate(self, aip: AIP) -> list[PREMISEvent]:
        events: list[PREMISEvent] = []
        sealed = self._archival_root / aip.aip_id
        shutil.copytree(aip.root, sealed, dirs_exist_ok=False)
        self._apply_object_lock(sealed)
        events.append(self._emit(PREMISEvent(
            event_type="ingestion", outcome="success",
            detail=f"sealed {aip.aip_id} under object-lock at {sealed}",
            linking_object=aip.aip_id, linking_agent="preservation-storage")))
        for replica_root in self._replica_roots:
            target = replica_root / aip.aip_id
            shutil.copytree(sealed, target, dirs_exist_ok=False)
            replica_digest = self._package_digest_of(target)
            outcome = "success" if replica_digest == aip.package_digest else "fail"
            events.append(self._emit(PREMISEvent(
                event_type="replication", outcome=outcome,
                detail=f"replica={target} digest={replica_digest}",
                linking_object=aip.aip_id, linking_agent="preservation-storage")))
            if outcome == "fail":
                raise PromotionError(f"replica digest mismatch at {target}")
        return events

    def _package_digest_of(self, root: Path) -> str:
        manifest = json.loads(
            (root / "tagmanifest-sha256.json").read_text(encoding="utf-8"))
        return hashlib.sha256(
            json.dumps(manifest, sort_keys=True).encode("utf-8")).hexdigest()

    def _apply_object_lock(self, path: Path,
                           retention_days: int = 3650) -> None:
        """Placeholder for the storage backend's WORM retention call."""
        marker = path / ".object-lock"
        marker.write_text(json.dumps({"retention_days": retention_days,
                                      "locked_at": _now()}), encoding="utf-8")


def run_promotion(sip: SIP, ledger_path: Path, staging: Path,
                  archival: Path, replicas: list[Path]) -> Optional[AIP]:
    orchestrator = PromotionOrchestrator(
        ledger=AuditLedger(ledger_path), staging_root=staging,
        archival_root=archival, replica_roots=replicas,
        allowed_puids=frozenset({"fmt/353", "fmt/354", "fmt/463"}))
    try:
        return orchestrator.promote(sip)
    except PromotionError as exc:
        logger.error(json.dumps({"promotion_failed": str(exc),
                                 "package_id": sip.package_id,
                                 "action": "quarantine"}))
        return None

The shape to notice is that the event ledger is not decoration — it is the return value that matters. Whether promotion succeeds or fails, the ledger holds an ordered, signed account of every gate, and that account is what makes the resulting AIP (or the quarantine decision) defensible. The detailed happy-path narration of this exact flow is the subject of promoting a validated SIP to AIP step by step.

# Compliance and Audit Requirements

ISO 16363 (the trustworthy-repository standard that formalized the earlier TRAC checklist) does not ask whether you have an OAIS pipeline; it asks whether every state change to a preservation object left a signed, timestamped, independently verifiable record. Promotion concentrates more such state changes than any other moment in the system, so it is where the event catalogue below must be complete. Each row maps a promotion gate to the OAIS functional entity it belongs to and the PREMIS event it must emit.

The design rule uniting the table is that the promotion code and the evidence are one path: a gate cannot advance the package without also emitting its event, so there is no code route that produces an AIP with a gap in its provenance. The auditor-facing translation of this table — what to show, in what order, to demonstrate a defensible promotion — is worked through in building a SIP-to-AIP compliance checklist.

# Operational Failure Modes and Mitigations

The promotion boundary is defined by how it fails, not by its happy path. The recurring failure modes below each have a mitigation an auditor expects to see implemented in the orchestrator, not merely written in a policy document. The governing principle is fail closed: when integrity cannot be proven, the boundary refuses to promote and raises an event.

The failure that most tests a repository’s discipline is the fixity mismatch after some files are already staged, because the tempting shortcut is to keep the good pages. Resisting that shortcut — rolling the entire package back to quarantine — is the whole subject of handling AIP promotion rollback on fixity failure.

# Configuration and Deployment Checklist

Promote a SIP in production only when every item below is verified in the target environment. Each maps to a gate in the orchestrator, an ISO 16363 criterion, or a failure mode above, and each should be backed by a monitoring hook rather than a one-time manual check.

• FIXITY_ALGORITHM set to sha256; boundary re-verification is independent of the ingestion-side check, not a reuse of it
• Manifest digests from the ingestion side are treated as claims to confirm, never as facts to trust
• ALLOWED_PUIDS pins the accepted format set; the signature file for DROID/Siegfried is current
• Declared-versus-detected format conflicts route to quarantine, never to a silent correction
• Dublin-Core-to-PREMIS mapping reviewed for lossless carry-over of identifiers, agents, and rights
• Stage-1 and stage-2 events are carried forward into the AIP’s PREMIS, not re-derived at packaging time
• AIP layout (BagIt or OAIS-native) produces a tag manifest with a per-file and a top-level digest
• ARCHIVAL_ROOT enforces WORM object-lock retention, verified unbreakable before sealing counts as done
• REPLICA_ROOTS lists at least one geographically distinct site; each replica is re-verified on write
• Promotion is idempotent on package_id; a retried run cannot create a second AIP
• Every gate emits its PREMIS event before its irreversible action, to an append-only, verifiable ledger
• PromotionError at any gate triggers quarantine-and-rollback with an alert, never a partial promotion
• Structured JSON logs aggregated with alerting on promotion_failed, fixityCheck fail, and ledger gaps

# Frequently Asked Questions

# Why re-verify fixity at promotion if the ingestion side already validated the batch?

Because the two checks are evidence of different things. The ingestion-side check confirms the files were intact when they were scanned and queued; the promotion-side check confirms they are still intact at the moment the repository takes durable custody, and it is performed by a different agent at a different time. A byte can rot or a transfer can truncate in the gap between the two. Independent re-verification at the boundary is what lets you assert — and prove to an auditor — that the object sealed into the AIP is byte-identical to the one the producer submitted.

# What exactly makes the SIP-to-AIP boundary the trust boundary?

Everything upstream of promotion is reversible without consequence: a batch can be rejected, re-scanned, or discarded and no preservation claim is broken. Promotion is the first irrevocable act — it commits the repository to holding the object, provably intact, for its full retention life. ISO 16363 assesses that durable claim, so the sequence of checks and events that produce the AIP is the concrete thing an auditor examines. If that sequence has an implicit or unlogged step, the claim is not defensible.

# If one file in a multi-file SIP fails a gate, can the rest still be promoted?

No. Promotion is atomic over the whole package. A 312-page survey that fails fixity on one page does not become an AIP with 311 good pages — it diverts to quarantine as a unit and rolls back any staged work. Splitting a failing package silently discards content while presenting the result as complete, which is exactly the kind of undocumented data loss an audit is designed to catch. The atomic-rollback behavior is handled explicitly in the rollback guide.

# Where does the ingestion pipeline actually hand off to the OAIS architecture?

At stage one. The ingestion area — scanning, routing, queuing, and schema validation — produces a SIP and asserts a set of digests via its validation schemas. The preservation area takes custody the instant the promotion boundary confirms those digests independently and begins emitting PREMIS events under its own agents. From fixity re-verification onward, every operation belongs to the OAIS side: format identification, PREMIS assembly, packaging, sealing, and replication are all preservation-architecture concerns.

# Related

• Automated Ingestion & Batch Scanning Workflows — the scanning, routing, and validation area that produces the SIP this walkthrough receives.
• OAIS-Compliant Digital Preservation Architecture — the preservation area that takes custody at the boundary and holds the sealed AIP.
• Promoting a Validated SIP to AIP, Step by Step — the happy-path narration of the orchestrator on this page.
• Building a SIP-to-AIP Compliance Checklist — the auditor-facing view of the promotion event catalogue.
• Handling AIP Promotion Rollback on Fixity Failure — the atomic quarantine-and-rollback path when a gate fails.
• Batch Validation Schemas — the upstream contract that produces the manifest digests the boundary re-verifies.