Handling AIP Promotion Rollback on Fixity Failure

Promoting a submission package into the archival tier is not a single atomic write — it is a sequence of independent side effects: bytes land on preservation storage, an AIP structure is registered, replicas fan out to secondary sites, and provenance events are emitted. When a fixity check fails partway through that sequence, the archive is left in a torn state that a naive delete cannot safely undo. This walkthrough, part of the End-to-End SIP-to-AIP Promotion guide, shows how to model promotion as a saga of compensating transactions so that a mismatched SHA-256 never leaves a half-sealed or corrupt AIP resident in the archival tier. It picks up exactly where promoting a validated SIP to an AIP step by step leaves off: that walkthrough assumes the happy path completes; this one governs what happens when the final integrity gate says no.

# Why a Failed Promotion Cannot Just Be Deleted

The instinct on a fixity mismatch is to delete whatever was written and try again. In a trustworthy repository that instinct is wrong for four concrete reasons, each rooted in the fact that promotion touches several systems that do not share a transaction boundary.

1. The write/verify gap. Bytes are streamed to storage and then re-read to compute a checksum. A crash, a truncated multipart upload, a silent bit flip on the storage fabric, or a race between the writer and the verifier all produce the same symptom — the digest the archive computes does not equal the digest the manifest recorded. At that moment some objects are on disk and some are not, and the process has no atomic rollback to fall back on.
2. Object-lock makes deletion impossible. Preservation storage is routinely configured with S3 Object Lock in compliance mode, WORM tape, or immutable filesystem snapshots. If a good copy of an object has already been sealed under a retention lock, the storage layer will refuse the delete outright. A rollback that assumes deletion always succeeds will itself fail, compounding the inconsistency.
3. Partial replicas. Replication to a second and third site is asynchronous. By the time fixity fails at the primary, one replica may already be durable, another may be mid-flight, and a third may not have started. Deleting only what the primary knows about strands orphaned copies at the other sites.
4. Events already emitted. By the point of failure the pipeline has very likely written a replication or messageDigestCalculation event into the provenance record. Provenance is append-only — you cannot retract history — so the rollback has to add a compensating event rather than erase the original.

Because these effects are non-transactional and partly irreversible, the correct primitive is not rollback-by-deletion but compensation: for every forward step, define an action that semantically undoes it, and on failure run those actions in reverse order.

# The Saga Model: Forward Steps and Their Compensations

A saga treats promotion as an ordered list of steps, each paired with a compensating action. The executor records every step that completes; on any failure it walks the completed steps backward and invokes each compensation. Sealing is the point of no return. Before the seal, compensations are true undo actions — retract the un-sealed writes, cancel in-flight replicas, discard staged bytes. After the seal, the archival copy is object-locked and authoritative, so its compensation is not deletion but supersession: mark the AIP version defective, promote a corrected version, and let retention policy expire the bad one. The diagram below traces both directions.

Forward promotion records a compensation for every step; a fixity mismatch runs those compensations in reverse to leave the archive consistent, while a post-seal failure supersedes the locked copy instead of deleting it.

Formally, the seal is permitted only when every stored object re-hashes to the digest the manifest promised:

$$\text{seal}(A) \iff \bigwedge_{f \in A} \operatorname{SHA256}\bigl(f_{\text{stored}}\bigr) = h_f$$

If a single conjunct is false, the saga must not seal — it must compensate.

# Step-by-Step: A Saga Executor with Reverse Compensation

The core is a small executor that remembers which forward steps completed and, on any exception, invokes their compensations in strict reverse order. Each compensation is logged so the rollback itself leaves an audit trail. Compensations must be idempotent and defensive: a compensation that assumes its forward step fully completed will itself throw when the failure happened during that step.

import logging
from dataclasses import dataclass, field
from typing import Callable

logger = logging.getLogger("archival.promotion")


@dataclass
class SagaStep:
    """One forward action paired with the action that semantically undoes it."""
    name: str
    forward: Callable[[], None]
    compensate: Callable[[], None]


class PromotionAborted(Exception):
    """Raised after compensation completes so callers know the AIP was rolled back."""


class PromotionSaga:
    """Executes promotion steps, compensating completed steps in reverse on failure."""

    def __init__(self, aip_id: str, sip_id: str) -> None:
        self.aip_id = aip_id
        self.sip_id = sip_id
        self._completed: list[SagaStep] = []

    def run(self, steps: list[SagaStep]) -> None:
        for step in steps:
            logger.info("saga_step_start", extra={"aip_id": self.aip_id, "step": step.name})
            try:
                step.forward()
            except Exception as exc:
                logger.error(
                    "saga_step_failed",
                    extra={"aip_id": self.aip_id, "step": step.name, "error": str(exc)},
                )
                self._compensate(failed_step=step.name, reason=str(exc))
                raise PromotionAborted(f"{self.aip_id} rolled back at {step.name}") from exc
            self._completed.append(step)
            logger.info("saga_step_ok", extra={"aip_id": self.aip_id, "step": step.name})

    def _compensate(self, failed_step: str, reason: str) -> None:
        logger.warning(
            "compensation_begin",
            extra={"aip_id": self.aip_id, "failed_step": failed_step,
                   "completed": [s.name for s in self._completed], "reason": reason},
        )
        for step in reversed(self._completed):
            try:
                logger.warning("compensation_run", extra={"aip_id": self.aip_id, "step": step.name})
                step.compensate()
                logger.info("compensation_ok", extra={"aip_id": self.aip_id, "step": step.name})
            except Exception as comp_exc:
                # A failing compensation is an operator-grade incident: the archive may be
                # inconsistent. Log CRITICAL and keep unwinding the remaining steps.
                logger.critical(
                    "compensation_failed",
                    extra={"aip_id": self.aip_id, "step": step.name, "error": str(comp_exc)},
                )

Wiring the promotion steps makes the pre-seal contract explicit. Notice that the fixity check is itself a step whose forward action raises on mismatch, and whose compensation is a no-op because it wrote nothing. The seal is the last step, so it never needs a delete-based compensation — anything that fails before it is retractable.

def build_promotion_saga(
    saga: PromotionSaga, storage: "ArchivalStorage", replicator: "Replicator",
    events: "EventLog", manifest: dict[str, str],
) -> list[SagaStep]:
    """Compose the ordered promotion steps and their compensating actions."""

    def stage_bytes() -> None:
        storage.stage(saga.aip_id, sealed=False)

    def discard_staged() -> None:
        # Un-sealed staging is not object-locked, so cleanup is a safe delete.
        storage.discard_unsealed(saga.aip_id)

    def write_components() -> None:
        storage.write_components(saga.aip_id, manifest)

    def retract_writes() -> None:
        storage.retract_unsealed(saga.aip_id)

    def register_replicas() -> None:
        replicator.register(saga.aip_id, sites=("site-b", "site-c"))

    def cancel_replicas() -> None:
        replicator.cancel(saga.aip_id)  # aborts in-flight, tombstones any that landed

    def verify_fixity() -> None:
        for path, expected in manifest.items():
            actual = storage.sha256(saga.aip_id, path)
            if actual != expected:
                events.emit(saga.aip_id, "fixityCheck", outcome="fail",
                            detail={"path": path, "expected": expected, "actual": actual})
                raise ValueError(f"fixity mismatch on {path}: {actual} != {expected}")

    def emit_failure_event() -> None:
        # Provenance is append-only: we add a compensating event, we never erase one.
        events.emit(saga.aip_id, "deletion", outcome="rolled-back",
                    detail={"sip_id": saga.sip_id, "reason": "fixity failure during promotion"})

    return [
        SagaStep("stage_bytes", stage_bytes, discard_staged),
        SagaStep("write_components", write_components, retract_writes),
        SagaStep("register_replicas", register_replicas, cancel_replicas),
        SagaStep("verify_fixity", verify_fixity, lambda: None),
        SagaStep("emit_success", lambda: events.emit(saga.aip_id, "ingestion", outcome="sealed"),
                 emit_failure_event),
    ]

When verify_fixity raises, the executor unwinds register_replicas (cancel), write_components (retract), and stage_bytes (discard) in that order, then the caller quarantines the SIP. The archive is returned to the state it held before promotion began, and a compensating deletion event records the reversal.

# Post-Seal Failure: Supersede, Never Delete

A fixity failure discovered after sealing — for example during a scheduled re-check, the subject of scheduling fixity re-validation jobs — cannot reuse the pre-seal compensations, because the object is under a retention lock. Deletion is both forbidden by the storage layer and wrong in principle: the defective version is part of the audit history. The compensation is to publish a corrected version and demote the old one.

def supersede_defective_aip(
    aip_id: str, corrected_manifest: dict[str, str],
    storage: "ArchivalStorage", events: "EventLog",
) -> str:
    """Handle a post-seal fixity failure without deleting the object-locked copy."""
    events.emit(aip_id, "fixityCheck", outcome="fail", detail={"phase": "post-seal"})
    new_aip_id = f"{aip_id}.v2"
    storage.stage(new_aip_id, sealed=False)
    storage.write_components(new_aip_id, corrected_manifest)
    # The old copy stays locked and readable; it is marked defective, not removed.
    storage.mark_superseded(aip_id, replaced_by=new_aip_id)
    events.emit(new_aip_id, "migration", outcome="supersedes",
                detail={"supersedes": aip_id, "reason": "post-seal fixity failure"})
    logger.warning("aip_superseded", extra={"old": aip_id, "new": new_aip_id})
    return new_aip_id

# Validation and Verification

Rollback logic is only trustworthy if you can prove it leaves nothing behind. Confirm a compensated promotion with the same evidence you would demand of any preservation action:

• Assert the archival tier is clean. After a forced mismatch, list the AIP prefix on primary storage and every replica site; there must be no un-sealed fragments and no orphaned replica objects. The cancel_replicas compensation should have tombstoned anything that raced to durability.
• Re-derive fixity from storage, not from logs. For every object the manifest named, recompute SHA-256 directly from what is on disk and confirm the mismatched object is absent rather than lingering. A log line saying “retracted” is not proof; the byte-level absence is.
• Inspect the provenance chain. The record must show the original fixityCheck failure and the compensating event (deletion with outcome: rolled-back, or migration with supersedes). Because provenance is append-only, both entries coexist — the failure is never edited away.
• Confirm the SIP is quarantined, not silently retried. The submission package should sit in a hold state for triage. Blind automatic retries against a deterministically corrupt master waste storage bandwidth; escalation belongs to the same discipline described in Error Handling & Retry Logic.

# Edge Cases and Gotchas

Three archival-specific traps deserve emphasis. Multi-part masters — a bound-volume TIFF written as several parts — can fail fixity on one part while the rest verify; retract the whole AIP, not just the bad part, or you seal an incomplete object. Legacy manifests carrying only MD5 must be treated as a legacy cross-check, never the audit-grade gate; compute SHA-256 on promotion and record both. Clock-skewed replica acknowledgements can make a replica look un-registered when it is in fact durable, so cancel_replicas should query each site authoritatively rather than trusting the primary’s local view.

# Frequently Asked Questions

# Why use a saga instead of a database transaction for AIP promotion?

Because promotion spans systems that do not share a transaction boundary: object storage, a replication service, and an append-only provenance log. No single BEGIN/COMMIT can wrap a multipart upload to S3, a fan-out to two other sites, and an event write. A saga accepts that reality — it lets each step commit independently and guarantees consistency by attaching a compensating action to every step, then running those compensations in reverse when a later step (here, the fixity check) fails.

# Can I just delete everything the failed promotion wrote?

Only for the un-sealed, un-locked artifacts. Once a good copy is sealed under Object Lock or written to WORM media, the storage layer will refuse the delete, and even if it did not, erasing a version that is already part of the audit history is a compliance violation. Pre-seal, deletion of staged bytes is the correct compensation; post-seal, the correct compensation is supersession — promote a corrected version and mark the defective one replaced.

# What happens to the provenance events that were already emitted?

They stay. Preservation provenance is append-only, so a rollback never edits or removes the original replication or fixityCheck entries. Instead the saga emits a compensating event — a deletion with an outcome of “rolled-back”, or a migration recording that a corrected AIP supersedes the defective one — so the full history, including the failure, remains auditable.

# What should happen to the SIP after a rollback?

Quarantine it. The submission package moves to a hold state for human or automated triage rather than being deleted or blindly re-queued. A deterministic fixity failure — a truncated source, a corrupt master — will fail identically on every retry, so unbounded automatic re-promotion only burns storage bandwidth. Escalation to a quarantine or dead-letter path is the disciplined response.

# Related

• Promoting a Validated SIP to an AIP Step by Step — the forward promotion walkthrough whose happy path this rollback logic protects.
• Scheduling Fixity Re-Validation Jobs — the recurring checks that surface post-seal fixity failures and trigger supersession.
• Error Handling & Retry Logic — quarantine and escalation patterns for the SIPs a rollback holds back from re-promotion.