Designing an Immutable Audit Ledger for Preservation Events

A trustworthy repository is judged not only by whether it performed a migration, a fixity check, or a normalization, but by whether the record of that work can itself be trusted years later. This page belongs to Preservation Action Logging and addresses the structural problem underneath every provenance claim: if the ledger that holds preservation events is mutable, uncorroborated, or unverifiable, then an ISO 16363 auditor cannot distinguish an authentic history from one silently rewritten after the fact. The concern here is the ledger itself — how records are linked, sealed, and independently re-checked — not the semantics of any single event. Emitting the events is the job of the sibling guide on emitting PREMIS events for preservation actions; this guide makes the container those events land in tamper-evident.

# Root-Cause Analysis of Untrustworthy Ledgers

An audit ledger fails to be evidence for one or more concrete reasons. Each is a discrete engineering defect, and each is independently sufficient to void the provenance record.

1. Mutable storage. A ledger kept in an ordinary relational table or a rewritable file offers no barrier to a privileged UPDATE, an fsync overwrite, or a compromised administrator editing history. Nothing in the storage layer distinguishes an authentic row from one changed yesterday, so the record proves only the current state, never the sequence of past states.
2. No cryptographic chaining. When records are independent rows, deleting or altering one leaves the others untouched and self-consistent. There is no internal structure that a tamper would violate, so no verifier can detect that record 4,102 was quietly rewritten.
3. Single copy. One authoritative ledger with no replicas means a single disk failure, ransomware event, or malicious wipe erases the entire chain of custody. Fixity on the archival objects is worthless if the record proving that fixity was ever checked is gone.
4. No independent verification. A ledger that only its own writer can validate is a ledger that reports on itself. Without a re-verification path a third party can execute against the raw stored bytes, the integrity claim reduces to “trust the operator” — exactly what a trustworthy-repository audit exists to eliminate.
5. Unbounded growth. A ledger that appends forever with no segmentation, rotation, or anchoring becomes too large to re-hash in reasonable time, so verification is quietly abandoned. An audit trail that is never checked is indistinguishable from one that was tampered.

The fix is to make each record cryptographically depend on every record before it, seal the head periodically with a signature written to write-once storage, and expose a walk any auditor can run. The diagram below shows the three moving parts together: the chain, the signed checkpoint anchoring the head, and the verifier that detects a tamper by recomputation.

Each record’s hash folds in the previous hash, a signed checkpoint seals the head, and an independent re-walk exposes any alteration and everything downstream of it.

# The Chaining and Checkpoint Relations

The core invariant is a Merkle-style linear chain. Let (r_n) be the (n)-th preservation event and (\text{canonical}(r_n)) its deterministic byte encoding. The chain begins from a fixed genesis sentinel (H_{-1}) and advances by folding each record into the digest of its predecessor:

$$H_n = \operatorname{SHA256}!\left(H_{n-1} ,\Vert, \operatorname{canonical}(r_n)\right)$$

Because (H_n) depends on (H_{n-1}), which depends on (H_{n-2}), and so on to the genesis, altering any (r_k) changes (H_k) and forces a different (H_n) for every (n > k). Tampering cannot stay local; it propagates to the head. A verifier confirms integrity by recomputing the whole chain and asserting equality at every position:

$$\forall, n:\quad H’_n \stackrel{?}{=} H_n, \qquad H’n = \operatorname{SHA256}!\left(H’ ,\Vert, \operatorname{canonical}(r_n)\right)$$

A single hash walk cannot, on its own, prove that records were never truncated — an attacker who controls storage could drop the tail and re-present a shorter but internally consistent chain. That gap is closed by periodically signing the head. A checkpoint (C_k) binds the current head hash, the record count (n), and a timestamp (t_k) under a private signing key:

$$C_k = \operatorname{Sign}_{sk}!\left(H_n ,\Vert, n ,\Vert, t_k\right)$$

Once (C_k) is written to write-once storage, the archive is committed to at least (n) records with that exact head. A later truncation or rewrite that produces a different head can no longer match a previously published, signed checkpoint, so the divergence is detectable by anyone holding the public key.

# Step-by-Step Resolution: An Append-Only Ledger

The implementation is a small, dependency-light Ledger that enforces the invariants above in code. Start with canonicalization and the link function, because deterministic bytes are the foundation — two encoders that disagree on key order or whitespace will compute different hashes for the same event and produce phantom tamper alarms.

from __future__ import annotations

import hashlib
import hmac
import json
import logging
from dataclasses import dataclass
from typing import Any

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger("archival.preservation.ledger")

GENESIS: str = "0" * 64  # fixed prev-hash sentinel H_{-1} for the first record


def canonical(record: dict[str, Any]) -> bytes:
    """Deterministic byte serialization so a record hashes identically everywhere."""
    return json.dumps(
        record, sort_keys=True, separators=(",", ":"), ensure_ascii=False
    ).encode("utf-8")


def link_hash(prev_hash: str, record: dict[str, Any]) -> str:
    """H_n = SHA-256(H_{n-1} || canonical(record)) as a hex digest."""
    digest = hashlib.sha256()
    digest.update(prev_hash.encode("ascii"))
    digest.update(canonical(record))
    return digest.hexdigest()

Each stored entry captures its sequence number, the hash it chained onto, the record, and its own hash. Freezing the dataclass communicates intent: entries are values, never mutated after they are appended.

@dataclass(frozen=True)
class Entry:
    seq: int
    prev_hash: str
    record: dict[str, Any]
    entry_hash: str

The ledger appends by reading the current head, computing the new link, and recording the entry. This is the only write path, and it is monotonic — there is no update and no delete. Every append emits a structured log line carrying the sequence, the previous hash, and the new head so the write is externally observable and correlatable with the PREMIS event it seals.

class Ledger:
    def __init__(self, signing_key: bytes) -> None:
        self._entries: list[Entry] = []
        self._signing_key = signing_key

    @property
    def head(self) -> str:
        return self._entries[-1].entry_hash if self._entries else GENESIS

    def append(self, record: dict[str, Any]) -> Entry:
        """Append one preservation event, chaining it onto the current head."""
        seq = len(self._entries)
        prev = self.head
        entry = Entry(seq=seq, prev_hash=prev, record=record,
                      entry_hash=link_hash(prev, record))
        self._entries.append(entry)
        logger.info(
            "ledger_append",
            extra={"seq": seq, "prev_hash": prev, "entry_hash": entry.entry_hash,
                   "event_type": record.get("eventType")},
        )
        return entry

Verification re-walks from genesis, recomputing each hash from the stored record bytes and asserting both that the stored prev_hash matches the running head and that the stored entry_hash matches the recomputed value. Either failure localizes the tamper to a sequence number and stops the walk.

class Ledger:  # continued from the append definition above
    def verify(self) -> bool:
        """Re-derive the whole chain from genesis and detect any alteration."""
        prev = GENESIS
        for entry in self._entries:
            recomputed = link_hash(prev, entry.record)
            if entry.prev_hash != prev:
                logger.error("ledger_break_prev", extra={"seq": entry.seq,
                             "expected_prev": prev, "stored_prev": entry.prev_hash})
                return False
            if entry.entry_hash != recomputed:
                logger.error("ledger_tamper", extra={"seq": entry.seq,
                             "recomputed": recomputed, "stored": entry.entry_hash})
                return False
            prev = entry.entry_hash
        logger.info("ledger_verified", extra={"count": len(self._entries), "head": prev})
        return True

Checkpointing signs the head. Production systems should prefer an asymmetric scheme such as Ed25519 so verifiers need only a public key, but a keyed HMAC over the same head || count || timestamp payload is a fully runnable stand-in that keeps the example dependency-free.

class Ledger:  # continued from the append definition above
    def checkpoint(self, timestamp: str) -> dict[str, str]:
        """Seal the current head under a signature; anchor this off-ledger and WORM."""
        head, count = self.head, len(self._entries)
        payload = f"{head}|{count}|{timestamp}".encode("ascii")
        signature = hmac.new(self._signing_key, payload, hashlib.sha256).hexdigest()
        checkpoint = {"head": head, "count": str(count),
                      "timestamp": timestamp, "signature": signature}
        logger.info("ledger_checkpoint", extra=checkpoint)
        return checkpoint

The checkpoint is only as durable as where it lands. Anchor it to write-once, read-many storage so no later process can overwrite the seal. On object storage, S3-style Object Lock in compliance mode gives WORM semantics that even the root account cannot shorten, and this is the same durability posture applied to archival packages themselves in Long-Term Storage Architecture.

import boto3


def persist_checkpoint_worm(bucket: str, key: str,
                            checkpoint: dict[str, str], retain_until: str) -> None:
    """Write a signed checkpoint under S3 Object Lock so it cannot be deleted or rewritten."""
    client = boto3.client("s3")
    client.put_object(
        Bucket=bucket, Key=key, Body=canonical(checkpoint),
        ObjectLockMode="COMPLIANCE", ObjectLockRetainUntilDate=retain_until,
    )
    logger.info("checkpoint_worm_persisted",
                extra={"bucket": bucket, "key": key, "retain_until": retain_until})

# Validation and Verification

A ledger design is only credible if a third party can reproduce its integrity claim from the raw bytes. Confirm the guarantees behaviorally, not by inspecting configuration:

• Re-walk from genesis. Run verify() against the stored entries and confirm it returns True for the count named in the latest checkpoint. This is the assertion an ISO 16363 auditor repeats independently, and it must succeed against a fresh process that shares no state with the writer.
• Inject a tamper and confirm detection. Alter one byte of any record and re-run verify(). It must return False and log ledger_tamper at the exact sequence, with every later record failing too — proof that the chain propagates alteration rather than localizing it.
• Reconcile against a signed checkpoint. Recompute the HMAC (or verify the Ed25519 signature) over the current head || count || timestamp and confirm it matches a previously WORM-persisted C_k. A mismatch or a head shorter than a signed count signals truncation.
• Cross-check against object provenance. Every ledger entry should correspond to a durable PREMIS event folded into object history — a discipline detailed in emitting PREMIS events for preservation actions — so the ledger and the objects’ own metadata corroborate each other rather than standing alone.
• Verify every replica identically. Run the same walk against each replica and confirm all report the same head. Replicas must carry the same fixity guarantees as archival packages; a divergent head is a split-brain history that demands investigation before any further append.

# Edge Cases and Gotchas

Two archival specifics deserve emphasis. First, segmentation is not merely an operational convenience — it is what keeps verification affordable at scale. Open each new segment with a genesis record whose payload is the signed checkpoint of the segment before it, so the segments themselves form a higher-order chain and a verifier can validate the current segment quickly while still trusting the sealed tail. Second, resist the temptation to fold access events into this same structure. Capturing who read or exported an object is a distinct concern with its own retention and privacy constraints, handled in auditing access events in an immutable ledger; this ledger stays focused on preservation actions so its schema and retention policy remain coherent.

# Frequently Asked Questions

# Why chain records cryptographically instead of using database constraints or triggers?

Database constraints and append-only triggers protect against accidental modification through the normal application path, but they are enforced by the same system an attacker with sufficient privilege controls. A SUPERUSER, a direct file edit, or a restored-from-backup rewrite bypasses every trigger. Hash chaining moves the guarantee out of the enforcement layer and into mathematics: the integrity of record (n) is checkable from the bytes alone, by anyone, without trusting the database that stored it. That is precisely the independence an ISO 16363 audit requires.

# How often should the ledger be checkpointed and signed?

Frequently enough that the window of undetectable truncation is acceptable, and always at segment boundaries. A common cadence is a signed checkpoint on every segment rotation plus a time-based one (hourly or daily) for active ledgers. Each checkpoint you publish to WORM storage is a commitment an attacker cannot walk back, so more frequent anchoring shrinks the tail they could ever drop. The cost is negligible — one signature over a fixed-size payload — so err toward more checkpoints rather than fewer.

# Does hash chaining alone make the ledger tamper-proof?

No. Chaining makes it tamper-evident, not tamper-proof: it guarantees that any alteration is detectable on re-verification, not that alteration is impossible. Two further controls close the remaining gaps. WORM backing storage prevents the head from being silently overwritten, and periodic signed checkpoints prevent the tail from being silently truncated. Chaining, WORM, and signed checkpoints together are what let a verifier assert that the history is both unaltered and complete.

# What belongs in a canonical record, and what must stay out?

Include the stable, semantically meaningful fields of the preservation event: event type, object identifier, agent, outcome, and the fixity digest computed at the time. Exclude anything non-deterministic or environment-specific — floating-point values, unordered sets, locale-formatted dates, or volatile handles — because any encoder disagreement on those fields produces a different hash and a false tamper alarm. Normalize timestamps to UTC in a fixed format and represent all values as strings or integers so the canonical bytes are reproducible on every platform that verifies the chain.

# Related

• Preservation Action Logging — the parent area covering how preservation events are recorded, sealed, and audited across the repository.
• Emitting PREMIS Events for Preservation Actions — how the events this ledger seals are produced and folded into object provenance.
• Long-Term Storage Architecture — the WORM, replication, and fixity posture the ledger’s checkpoints inherit from archival package storage.