Migrating AIPs Between Cold-Storage Tiers Safely

Every long-lived archive eventually re-homes its bits: a vendor deprecates a tier, tape reaches end of media life, egress economics push a collection from Amazon S3 Glacier Deep Archive to an on-premise object store, or a trustworthy-repository audit demands geographic redundancy. This page is the migration reference for the Long-Term Storage Architecture stage: it defines exactly how to move an Archival Information Package between cold-storage tiers or providers — Glacier to tape, tape to another object store, one region to another — without ever losing integrity or deleting the only good copy. The specific failure it prevents is the one that never shows up in a dashboard: a migration that reports success while the destination holds a truncated, re-encoded, or silently corrupted object, and the source has already been released under a retention clock the operator forgot was ticking.

Cold storage makes this uniquely dangerous. Unlike hot object storage, a Glacier or tape tier will not let you cheaply read back what you just wrote to confirm it landed intact; a restore can take hours and cost money, so the temptation is to trust the write and delete the source. That single shortcut — trusting a write you never independently read — is the root of almost every cold-tier migration disaster.

# Root Causes of Migration Data Loss

Four concrete mechanisms account for nearly all integrity loss during a tier-to-tier move. Each is silent by default; none raises an exception unless you go looking.

1. Delete-before-verify. The migration copies the object, receives a 200 OK or a successful PUT response, and immediately deletes or lifecycles the source. But an HTTP success confirms only that the request was accepted — not that the destination now holds a byte-identical AIP. If the copy was partial or the storage layer later fails a durability check, the source is already gone and the AIP is unrecoverable.

2. Silent truncation on transfer. Streaming a multi-gigabyte AIP through a broken pipe, a connection reset, or a multipart upload that completes with a missing part yields an object that is shorter than the original but structurally valid enough to store. Object stores happily accept a 4 GB upload where 6 GB was intended if the client finalized the multipart manifest. Nothing about the stored object announces that two gigabytes are missing.

3. Object-lock and retention masking the failure. Write-Once-Read-Many locks (S3 Object Lock, Glacier Vault Lock) are essential for preservation security, but during migration they cut both ways. A retention lock on the destination can silently reject an overwrite meant to repair a bad copy, leaving the flawed object in place. A retention lock on the source can block the release you scheduled, so a naïve migration marks the object “migrated” and moves on while the source lingers — or, worse, the operator forcibly overrides the lock to “finish” and destroys the safety copy.

4. Checksum recomputed from a cached copy, not the destination read. The most insidious failure: the verification step re-hashes the local buffer, the source object, or a CDN/cache-fronted read instead of performing an independent read from the destination tier. The digest matches — because it is hashing the wrong bytes. The destination is never actually verified, and the whole protocol collapses into an expensive no-op that produces a green checkmark over corrupted storage.

The through-line is that a successful write and a verified object are different claims. Closing that gap requires reading the destination back through the cold tier itself and recomputing the digest from those bytes before any release is even considered.

# The Safe-Migration State Machine

The protocol is deliberately linear and one-directional: the source AIP is never released until the destination has been independently re-read and its recomputed digest matches the source of truth. Any mismatch routes to rollback-and-alert, leaving the source authoritative and intact. The diagram below is the exact control flow the code implements.

The source AIP is released only after an independent destination re-read reproduces its recorded digest; any mismatch rolls back the copy and keeps the source authoritative.

# Step-by-Step Resolution: Verify, Copy, Verify, Then Release

The protocol has four ordered guarantees. Each step exists to close one of the root causes above, and the ordering is not negotiable — releasing the source before an independent destination verification reintroduces delete-before-verify no matter how careful the copy was.

Steps 3 and 4 carry the weight. The re-read must go through the cold tier’s real read path — for Glacier, that means a restore/retrieval, not a HEAD; for tape, a genuine read of the written block, not the staging cache. If independently reading the destination is prohibitively expensive for every object, the correct compromise is to verify a server-computed checksum the storage provider guarantees is derived from the stored bytes (for example, S3’s checksum algorithms or x-amz-checksum-sha256), never a client-side value echoed back. The one thing you must never do is hash the source or the local buffer a second time and call it destination verification.

The guarded migration function below implements the full protocol with structured logging at every transition and a PREMIS event emitted on the terminal outcome. It is deliberately conservative: mark_source_releasable is reached only after the destination digest is asserted equal to the recorded source digest.

from __future__ import annotations

import hashlib
import logging
from dataclasses import dataclass, field
from datetime import datetime, timezone
from typing import Callable, Protocol

logger = logging.getLogger("archival.migration")

# 64 KiB read window keeps memory bounded on multi-gigabyte AIPs.
_CHUNK = 64 * 1024


class StorageTier(Protocol):
    """Minimal contract each cold-storage backend must satisfy."""

    name: str

    def put(self, key: str, reader: Callable[[], bytes]) -> int:
        """Store the object; return the byte count the backend recorded."""

    def open_read(self, key: str) -> "SupportsRead":
        """Open an INDEPENDENT read stream from durable storage.

        Implementations must read from the stored object itself (a Glacier
        retrieval, a real tape read), never a write-side cache or buffer.
        """

    def delete(self, key: str) -> None:
        """Remove an object (used to roll back a failed destination copy)."""

    def retention_active(self, key: str) -> bool:
        """True while an object-lock / retention window forbids deletion."""


class SupportsRead(Protocol):
    def read(self, size: int) -> bytes: ...


@dataclass(frozen=True)
class MigrationEvent:
    """A PREMIS-aligned replication/migration event with both digests."""

    event_type: str            # "migration" (or "replication" for copies kept)
    outcome: str               # "success" | "failure"
    source_tier: str
    target_tier: str
    object_key: str
    source_digest: str
    destination_digest: str
    byte_length: int
    timestamp: str = field(
        default_factory=lambda: datetime.now(timezone.utc).isoformat()
    )


def _sha256_of_stream(stream: SupportsRead) -> tuple[str, int]:
    """Digest a stream chunk-by-chunk; return (hex_digest, byte_length)."""
    digest = hashlib.sha256()
    total = 0
    while True:
        block = stream.read(_CHUNK)
        if not block:
            break
        digest.update(block)
        total += len(block)
    return digest.hexdigest(), total


class MigrationError(RuntimeError):
    """Raised when the destination cannot be verified; source is preserved."""


def migrate_aip(
    object_key: str,
    source: StorageTier,
    destination: StorageTier,
    recorded_source_digest: str,
    recorded_byte_length: int,
    emit_event: Callable[[MigrationEvent], None],
    mark_source_releasable: Callable[[str], None],
) -> MigrationEvent:
    """Copy one AIP to a new tier, verify it independently, then release the source.

    The source is released ONLY after the destination is re-read from durable
    storage and its recomputed digest equals ``recorded_source_digest``. Any
    failure rolls back the destination copy and leaves the source authoritative.
    """
    ctx = {
        "object_key": object_key,
        "source_tier": source.name,
        "target_tier": destination.name,
    }
    logger.info("migration_start", extra={**ctx, "expected_bytes": recorded_byte_length})

    # --- Step 2: copy, streaming from the source's durable read path. ---
    def _source_reader() -> bytes:
        return source.open_read(object_key).read(_CHUNK)

    written = destination.put(object_key, _source_reader)
    if written != recorded_byte_length:
        destination.delete(object_key)  # rollback truncated copy
        logger.error(
            "migration_truncation",
            extra={**ctx, "expected_bytes": recorded_byte_length, "written_bytes": written},
        )
        raise MigrationError(
            f"Truncated copy: wrote {written} of {recorded_byte_length} bytes"
        )
    logger.info("migration_copied", extra={**ctx, "written_bytes": written})

    # --- Step 3: INDEPENDENT destination re-read + recompute. ---
    with_stream = destination.open_read(object_key)
    dest_digest, dest_bytes = _sha256_of_stream(with_stream)
    logger.info(
        "migration_reverify",
        extra={**ctx, "destination_digest": dest_digest, "destination_bytes": dest_bytes},
    )

    if dest_digest != recorded_source_digest:
        destination.delete(object_key)  # rollback corrupted copy
        event = MigrationEvent(
            event_type="migration",
            outcome="failure",
            source_tier=source.name,
            target_tier=destination.name,
            object_key=object_key,
            source_digest=recorded_source_digest,
            destination_digest=dest_digest,
            byte_length=dest_bytes,
        )
        emit_event(event)
        logger.error("migration_digest_mismatch", extra={**ctx, "event": event.outcome})
        raise MigrationError(
            f"Digest mismatch: destination {dest_digest} != source {recorded_source_digest}"
        )

    # --- Step 4: release the source, but never override active retention. ---
    if source.retention_active(object_key):
        logger.warning(
            "migration_release_deferred",
            extra={**ctx, "reason": "source retention/object-lock active"},
        )
    else:
        mark_source_releasable(object_key)
        logger.info("migration_source_releasable", extra=ctx)

    event = MigrationEvent(
        event_type="migration",
        outcome="success",
        source_tier=source.name,
        target_tier=destination.name,
        object_key=object_key,
        source_digest=recorded_source_digest,
        destination_digest=dest_digest,
        byte_length=dest_bytes,
    )
    emit_event(event)
    logger.info("migration_success", extra={**ctx, "destination_digest": dest_digest})
    return event

Notice that mark_source_releasable is called only inside the success path, and even then only when retention_active returns false — the function defers rather than forces release when a lock is present, which is the behavior that keeps a WORM policy from being trampled to “finish” a migration. The mismatch and truncation branches both delete the destination copy and preserve the source, so a failed migration is always recoverable by re-running it.

# Validation and Verification

Confirming the migration is not the same as confirming the function returned. Prove the move with evidence an auditor can inspect:

• Assert the digest equality out of band. After a batch, re-read a sample of migrated objects directly from the destination tier and recompute SHA-256, comparing against the recorded source digest — the same independent read the protocol performs, run again by a different process. Because the digest was computed from a destination read, a passing check certifies the stored bytes, not a cache.
• Inspect the PREMIS event, not the log line. Every terminal outcome emits a MigrationEvent carrying both source_digest and destination_digest, the two tiers, and the outcome. Folding these into object provenance through the preservation action log gives a durable record that the copy was verified before release, which is exactly what an ISO 16363 auditor asks to see.
• Confirm the source still exists on any failure. For every MigrationError, verify the source object is intact and the destination copy was removed. A failed migration must leave the archive in its pre-migration state, never in a half-deleted limbo.
• Schedule a post-migration fixity sweep. The move changes where the bits live and resets the storage medium’s age clock, so re-baseline fixity on the new tier and let scheduled fixity re-validation carry the object forward on its normal cadence.

# Edge Cases and Gotchas

• Glacier retrieval latency vs. verification budget. An independent re-read from Deep Archive can take twelve hours and incur retrieval fees. For very large collections, verify a provider-guaranteed server-side checksum (x-amz-checksum-sha256) that is derived from the stored object, and reserve full-object re-reads for a statistical sample plus every object whose server checksum is absent. Never substitute the checksum the client sent for one the storage computed.
• Multipart uploads that “succeed” with a missing part. Object stores finalize a multipart upload from the part manifest; a dropped final part can yield a shorter object that still completes. The byte-length guard in step 2 catches this, which is why the recorded length must come from the source of truth, not from the upload response.
• Server-side copy that never leaves the provider. A same-provider CopyObject between tiers may be a metadata operation that shares underlying storage, so it does not give you an independent second copy for durability purposes. Treat cross-tier moves within one provider as still requiring an independent re-read, and prefer genuine cross-provider copies when the goal is geographic or vendor redundancy.
• Retention on the destination blocking a repair. If a first migration wrote a bad object under an object-lock, the retention window can forbid the overwrite the rollback needs. Write to a fresh key (versioned or timestamped) rather than reusing the locked key, so a corrupted destination object can be abandoned rather than fought.
• Digest algorithm drift. Migrating an AIP whose original fixity was recorded in MD5 (legacy cross-check only) is the moment to compute and record a SHA-256 alongside it, so the destination becomes the first copy carrying an audit-grade digest. Record both in the event; never silently downgrade to the weaker algorithm to make a comparison pass.

For the tiering policy that decides which objects move and when, see best practices for cold-storage tiering; this page assumes that decision is already made and focuses only on executing the move without loss. Canonical event semantics follow the Library of Congress PREMIS Data Dictionary, and the digest math relies on the standard-library hashlib module.

# Frequently Asked Questions

# Why re-read the destination instead of trusting the copy’s success response?

Because a successful write response confirms only that the request was accepted, not that the destination holds byte-identical data. Partial multipart uploads, truncated streams, and latent storage corruption all produce a “successful” write over a flawed object. Independently reading the destination back through the cold tier’s real read path and recomputing the digest is the only way to prove the stored bytes match the source before you release the original.

# Can I recompute the checksum locally to save a retrieval?

Only if you hash the destination’s bytes. The failure mode this protocol exists to prevent is hashing the source, the local buffer, or a cached read and calling it destination verification — the digest matches because it is hashing the wrong data. If a full destination re-read is too costly, verify a checksum the storage provider computed from the stored object (such as S3’s server-side SHA-256), never a value the client supplied and the server merely echoed.

# What happens to the source AIP if verification fails?

Nothing — that is the point. On a byte-count shortfall or a digest mismatch, the function deletes the flawed destination copy, emits a failure event carrying both digests, raises MigrationError, and never calls mark_source_releasable. The source remains the authoritative copy and the migration can simply be retried once the cause is fixed.

# How does object-lock or retention interact with releasing the source?

Retention is honored, never overridden. Even on a verified match, the function checks retention_active on the source and defers release while a WORM lock or retention window is in force, logging a deferral rather than forcing a deletion. Releasing an AIP by breaking its own retention policy to “complete” a migration is itself a preservation-security violation, which is why the release is conditional.

# Related

• Long-Term Storage Architecture — the parent stage: tier topology, durability targets, and where migration sits in the storage lifecycle.
• Best Practices for Cold-Storage Tiering — the policy that decides which AIPs live on which tier and when a move should be triggered.
• Scheduling Fixity Re-Validation Jobs — the ongoing fixity cadence that re-baselines an AIP after it lands on a new tier.