Scheduling Fixity Re-Validation Jobs

A checksum recorded once at ingest proves only that a file was intact the moment it entered the archive. Silent bit rot — a flipped bit on a decaying LTO cartridge, an undetected read error on a spun-down drive, a firmware bug that corrupts a block on write-back — accumulates in storage that no one ever reads again, precisely because preservation masters are, by design, rarely accessed. This page belongs to the Preservation Action Logging area and covers the recurring job that closes that gap: a scheduler that periodically re-reads objects, recomputes their digests, compares against the value of record, and turns every comparison — pass or fail — into a durable fixityCheck event. The engineering problem is not whether to re-validate but how often, because re-reading every object on every cycle is prohibitively expensive on cold media, while re-reading nothing lets corruption sit undetected until the day someone finally needs the file and discovers it is gone.

# Root-Cause Analysis of Undetected Corruption

Five distinct failure patterns let bit rot go unnoticed in an otherwise well-run repository. Each maps to a specific scheduling decision the sections below correct.

1. Fixity is calculated once and never revisited. The messageDigestCalculation event from ingest is treated as permanent proof. Storage media degrades on a timescale of years; a digest computed in 2019 says nothing about the bytes on disk in 2026. Without a recurring job there is no mechanism that would ever notice a flipped bit.
2. Everything is re-read too often. The over-correction is a nightly full sweep of the entire holdings. On a cold tier this is ruinous: spinning up archived LTO tapes or waking cold-line cloud objects incurs retrieval cost, and repeated reads accelerate media wear and thermal cycling — the very failure the check is meant to catch. Verification effort must be proportional to risk, not uniform.
3. No prioritization by tier or age. A single global cadence treats a hot SSD replica and a decade-old tape identically. The tape is far more likely to have degraded and far more expensive to read, so it needs a different sampling strategy — not the same one applied blindly everywhere.
4. Failures are logged but not turned into events. A mismatch that only produces a stack trace or a log line is not actionable and is not auditable. Under OAIS and ISO 16363, a fixity failure is a first-class preservation event that must be recorded against the object, alongside the passes, so the provenance record shows a continuous chain of verification rather than a single stale digest.
5. Jobs are not resumable. A sweep that restarts from the beginning after a crash either never finishes a large tier or repeats work it already did, wasting exactly the cold-media reads that are the scarce resource. Without a persisted cursor there is no guarantee the whole object set is covered within the intended window.

The corrective architecture is a risk-based cadence per storage tier driving a rolling full-verification window, executed by a resumable scheduler that emits an event per object and quarantines any mismatch. The diagram below shows the loop.

The scheduling loop: risk-based selection feeds recompute-and-compare; a match advances the rolling cursor, a mismatch becomes a recorded event plus an alert and quarantine.

# Coverage, Windows, and Expected Time to Detect

The scheduling decision reduces to a single trade-off: how much of a tier you verify per unit time versus how long a corruption can sit undetected. Model each tier independently. Let (N_{\text{tier}}) be the object count and (r_{\text{tier}}) the sustainable re-validation rate (objects per day) that respects the tier’s retrieval budget and wear limits. The full-verification window — the time to cover every object once — and the expected latency before a randomly-timed corruption is caught under uniform rolling verification are:

$$W_{\text{tier}} = \frac{N_{\text{tier}}}{r_{\text{tier}}}, \qquad \mathbb{E}[T_{\text{detect}}] = \frac{W_{\text{tier}}}{2} = \frac{N_{\text{tier}}}{2,r_{\text{tier}}}$$

Doubling the rate halves both the window and the expected detection latency, at double the read cost. For a hot tier where reads are cheap you drive (W) down to weeks; for a deep-cold tape tier you accept a window of months and supplement it with sampling. If instead of a strict rolling cursor you draw a random sample of fraction (f) each scheduling interval (\Delta t), detection of a given corrupt object follows a geometric process and the expected latency is:

$$\mathbb{E}[T_{\text{sample}}] = \frac{\Delta t}{f}$$

Sampling gives probabilistic early warning of a systemic problem — a failing tape library or a bad controller batch shows up as clustered mismatches long before a strict cursor would reach the affected objects — while the rolling window guarantees eventual total coverage. Production schedules run both: a rolling cursor for the coverage guarantee, plus a random sample overlay for fast systemic detection.

# Step-by-Step Resolution

# Define per-tier cadence and a resumable cursor

Encode the risk-based cadence as data, not as buried constants, so policy can change without touching the executor. Each tier declares its full-verification window and a sampling fraction; the scheduler translates the window into a due-date per object. The cursor persists the last object verified per tier so a crashed sweep resumes exactly where it stopped rather than re-reading cold media from the top.

from __future__ import annotations

import logging
from dataclasses import dataclass
from datetime import datetime, timedelta, timezone

logger = logging.getLogger("archival.fixity.scheduler")


@dataclass(frozen=True)
class TierPolicy:
    """Risk-based re-validation cadence for one storage tier."""

    name: str
    full_window_days: int   # cover every object at least once within this window
    sample_fraction: float  # extra random overlay per run, for systemic detection


TIER_POLICIES: dict[str, TierPolicy] = {
    "hot":  TierPolicy("hot",  full_window_days=30,  sample_fraction=0.02),
    "warm": TierPolicy("warm", full_window_days=120, sample_fraction=0.01),
    "cold": TierPolicy("cold", full_window_days=365, sample_fraction=0.005),
}


@dataclass
class ArchivalObject:
    object_id: str
    tier: str
    storage_path: str
    expected_digest: str          # value of record (SHA-256, audit-grade)
    last_verified: datetime | None


def is_due(obj: ArchivalObject, policy: TierPolicy, now: datetime) -> bool:
    """An object is due when its window has elapsed since the last check."""
    if obj.last_verified is None:
        return True
    deadline = obj.last_verified + timedelta(days=policy.full_window_days)
    return now >= deadline

# Select the due set by tier cadence

The selection query is the whole risk model in action: it returns objects whose window has elapsed, ordered by how overdue they are so the most at-risk objects go first, and it caps the batch at each tier’s sustainable rate so a backlog never triggers a cold-tier stampede. The cap is (r_{\text{tier}} = N_{\text{tier}} / W_{\text{tier}}) from the formula above, expressed per run.

from collections.abc import Iterable


def select_due_objects(
    catalog: Iterable[ArchivalObject],
    now: datetime,
    runs_per_day: int = 1,
) -> list[ArchivalObject]:
    """Return objects due for re-validation, rate-capped per tier."""
    by_tier: dict[str, list[ArchivalObject]] = {}
    for obj in catalog:
        policy = TIER_POLICIES.get(obj.tier)
        if policy is None:
            logger.warning("unknown_tier", extra={"object_id": obj.object_id, "tier": obj.tier})
            continue
        if is_due(obj, policy, now):
            by_tier.setdefault(obj.tier, []).append(obj)

    selected: list[ArchivalObject] = []
    for tier, objects in by_tier.items():
        policy = TIER_POLICIES[tier]
        # Sustainable objects-per-run = tier size / window / runs-per-day.
        per_run_cap = max(1, len(objects) // max(1, policy.full_window_days * runs_per_day))
        objects.sort(key=lambda o: o.last_verified or datetime.min.replace(tzinfo=timezone.utc))
        capped = objects[:per_run_cap]
        selected.extend(capped)
        logger.info(
            "tier_selection",
            extra={"tier": tier, "due": len(objects), "selected": len(capped),
                   "window_days": policy.full_window_days},
        )
    return selected

# Recompute, compare, and emit a fixityCheck event per object

The worker re-reads the object from its tier, recomputes the digest, and compares it against the value of record. Every comparison emits a fixityCheck PREMIS event carrying both the expected and the observed digest and an outcome of pass or fail — this is the same event vocabulary produced by emitting PREMIS events for preservation actions, so re-validation results fold into the same audit trail as ingest and migration events. A mismatch is logged at error level, raises an alert, and quarantines the object rather than silently overwriting the record.

import hashlib
from typing import Literal, Protocol

Outcome = Literal["pass", "fail"]


class EventSink(Protocol):
    def emit(self, event: dict[str, object]) -> None: ...


def compute_sha256(path: str, chunk_size: int = 1 << 20) -> str:
    """Stream the object off storage so a large master never loads into memory."""
    digest = hashlib.sha256()
    with open(path, "rb") as handle:
        for block in iter(lambda: handle.read(chunk_size), b""):
            digest.update(block)
    return digest.hexdigest()


def build_fixity_event(obj: ArchivalObject, observed: str, outcome: Outcome,
                       now: datetime) -> dict[str, object]:
    """A PREMIS fixityCheck event recording expected vs observed digest."""
    return {
        "eventType": "fixityCheck",
        "eventDateTime": now.isoformat(),
        "linkingObjectIdentifier": obj.object_id,
        "eventOutcome": outcome,
        "eventDetail": {
            "algorithm": "SHA-256",
            "expectedDigest": obj.expected_digest,
            "observedDigest": observed,
            "storageTier": obj.tier,
        },
    }


def revalidate(obj: ArchivalObject, sink: EventSink,
               quarantine: EventSink, now: datetime) -> Outcome:
    """Recompute the digest, compare, emit an event, and quarantine on mismatch."""
    observed = compute_sha256(obj.storage_path)
    outcome: Outcome = "pass" if observed == obj.expected_digest else "fail"
    event = build_fixity_event(obj, observed, outcome, now)
    sink.emit(event)

    if outcome == "pass":
        obj.last_verified = now
        logger.info("fixity_pass",
                    extra={"object_id": obj.object_id, "tier": obj.tier})
    else:
        logger.error(
            "fixity_mismatch",
            extra={"object_id": obj.object_id, "tier": obj.tier,
                   "expected": obj.expected_digest, "observed": observed},
        )
        quarantine.emit({"object_id": obj.object_id, "reason": "fixity_mismatch",
                         "observed": observed, "detected": now.isoformat()})
    return outcome

# Wire it to a beat scheduler

select_due_objects runs on a periodic trigger and enqueues one revalidate job per selected object. Under Celery beat this is a scheduled entry; under system cron it is a management command. Because selection is idempotent and the cursor is last_verified, a crash mid-run simply leaves those objects due next time — nothing is skipped and nothing is double-read within a window.

def run_scheduled_sweep(catalog: list[ArchivalObject], sink: EventSink,
                        quarantine: EventSink) -> dict[str, int]:
    """One scheduler tick: select the due set and re-validate each object."""
    now = datetime.now(timezone.utc)
    due = select_due_objects(catalog, now)
    tally = {"pass": 0, "fail": 0}
    for obj in due:
        outcome = revalidate(obj, sink, quarantine, now)
        tally[outcome] += 1
    logger.info("sweep_complete",
                extra={"checked": len(due), "passed": tally["pass"], "failed": tally["fail"]})
    return tally

# Validation and Verification

Confirm the scheduler behaves before you trust it with the only copies that matter:

• Deliberate corruption. In a staging archive, flip a byte in a warm-tier object with a hex editor, run run_scheduled_sweep, and confirm a fixityCheck event with eventOutcome: fail is emitted, the object lands in quarantine, and the observed digest in the event differs from the expected one exactly as the edit predicts.
• Coverage audit. After a full window has elapsed, assert that min(last_verified) across a tier is no older than full_window_days — the guarantee the rolling window is supposed to provide. If it is older, the per-run cap is starving that tier and (r_{\text{tier}}) must rise.
• Resumability. Kill the sweep process mid-run and restart it; verify no object is re-read twice within its window and that previously-unchecked objects are still selected — proof the cursor, not a restart-from-top, drives selection.
• Event, not log line. Query the provenance store, not the application log, for the day’s fixityCheck events. The count of passes plus fails must equal the number of objects the sweep reported checking, confirming every comparison became a durable, auditable event.

# Edge Cases and Gotchas

The recovered object itself is a separate workflow: quarantine isolates the file and raises the alert, but restoring from a replica or a different tier is a storage operation covered by long-term storage architecture and, when the good copy lives on another tier, by migrating AIPs between cold storage tiers safely. Fixity re-validation detects and records; repair is deliberate and separately audited.

# Frequently Asked Questions

# How often should I re-validate objects on cold tape versus a hot replica?

Set the cadence from the tier’s cost and risk, not a single global number. A hot replica where reads are cheap can carry a full-verification window of weeks; a deep-cold tape tier should run a window of months because every read costs retrieval time and adds media wear. From the model above, a tier’s per-run rate is its object count divided by its window, so declaring the window per tier and letting the scheduler derive the rate keeps effort proportional to risk automatically.

# Why sample randomly if a rolling cursor already covers everything eventually?

Coverage and early warning are different guarantees. The rolling cursor promises that every object is checked within its window, but it reaches objects in a fixed order, so a systemic failure — a dying tape library, a bad controller batch — might not surface until the cursor happens to arrive there. A small random sample overlay draws from the whole tier every run, so clustered mismatches from a systemic fault show up fast, giving expected detection latency of the interval divided by the sample fraction rather than half the full window.

# What should happen the moment a digest does not match?

Three things, in order: emit a fixityCheck event with eventOutcome: fail carrying both the expected and observed digests, quarantine the object so nothing downstream treats it as trustworthy, and alert an operator. What must not happen is an automatic overwrite of the recorded digest — that would erase the evidence of corruption. Repair is a deliberate, separately-audited restore from a known-good copy.

# Does re-reading objects to verify them accelerate the media decay I am trying to catch?

It can, which is exactly why uniform full sweeps are the wrong design for cold media. Reads add wear and, for tape, mechanical and thermal cycling. The risk-based cadence limits this by reading cold tiers infrequently and leaning on a small sample plus the coverage window, so total reads stay within the tier’s wear and cost budget while still guaranteeing every object is checked before its window expires.

# Related

• Preservation Action Logging — the parent area covering how preservation actions become durable, auditable events.
• Emitting PREMIS Events for Preservation Actions — the event vocabulary and structure that fixity results are recorded in.
• Migrating AIPs Between Cold Storage Tiers Safely — the storage workflow that restores or relocates an object after a mismatch is quarantined.