Validating OCR Confidence Before AIP Promotion

A text layer is the one part of a digitized document that looks finished long before it is correct. The characters render, the PDF is searchable, the derivative passes a visual spot-check — and yet the underlying recognition may be thirty percent noise that no one will notice until a researcher’s full-text query silently misses the record a decade from now. This guide is the confidence-gating reference for the OCR Processing Pipelines stage of the broader ingest workflow: it shows how to measure recognition quality quantitatively and make a machine-checkable promotion decision, so that a low-quality text layer is caught at the gate and never sealed into an archival package. The specific failure it solves is the irreversibility of promotion — once a text layer is fixed into an Archival Information Package (AIP), checksummed, and written to preservation storage, it is provenance, and correcting it means a fresh capture, a new normalization, and a migration event. The cheap moment to reject bad OCR is before that seal, not after.

# Root-Cause Analysis of Bad Text Layers

Low mean confidence is a symptom; promotion gating is only useful if it routes each failure back to the correct upstream cause. Five recurring root causes account for nearly every batch that a confidence gate should reject:

1. Insufficient capture resolution. Tesseract’s LSTM engine is trained around roughly 300 DPI for body text. Masters captured at 150–200 DPI — common for legacy microfilm scans or oversized bound volumes flattened to fit a platen — starve the recognizer of glyph detail, and confidence collapses on small type and diacritics long before a human notices blur.
2. Uncorrected skew. A page rotated even 2–3 degrees breaks the horizontal line-finding that page-segmentation depends on. Words fragment across the deskew boundary, baselines wander, and per-word confidence craters in bands rather than uniformly — a fingerprint that distinguishes a skew problem from a resolution problem.
3. Bleed-through and show-through. Thin or acidic paper lets the verso image ghost onto the recto. The binarizer promotes that ghost to foreground ink, and the engine confidently recognizes phantom characters — a case where mean confidence can look acceptable while the low-confidence word ratio spikes, which is why a gate must measure both.
4. Wrong language model. Running the default eng model over Fraktur, historical orthography, a diacritic-heavy Latin script, or mixed-language material forces the recognizer to map real glyphs onto the nearest wrong character. This is the most under-diagnosed cause because the output is fluent-looking garbage with deceptively mid-range scores.
5. Degraded originals. Foxing, water damage, faded iron-gall ink, and bleaching raise the noise floor of the image itself. No engine parameter recovers information the paper no longer carries; these pages are the legitimate case for routing to human transcription rather than re-scanning.

Each cause leaves a different signature in the confidence distribution — a uniform depression, a banded collapse, a bimodal split, or a spike in the low-confidence tail — so the gate below computes several complementary metrics rather than a single average, and the routing table later in this guide maps each signature back to its remedy.

# The Promotion Gate

The gate is a decision function, not a threshold check. Recognition produces a per-word confidence stream; the gate reduces that stream to page-level and batch-level metrics, compares them against a policy, and emits exactly one of two outcomes — promote toward AIP sealing, or quarantine and requeue for re-scan or manual QA. Every path also emits an OCR-quality preservation event so the decision itself becomes part of the object’s provenance. The flow below is the contract the code in the next section implements.

The gate reduces per-word confidence to a small metric set, promotes only on a full pass, and records an OCR-quality preservation event on every outcome so the decision itself is auditable.

# Step-by-Step Resolution: Computing Metrics and the Gate Decision

Tesseract’s tsv configuration emits one row per recognized token with a conf column scored 0–100 (and -1 for structural rows that carry no text). That stream is the raw material for every metric. The parser below reads the TSV, discards structural and empty rows, and yields clean (text, confidence) pairs. Keeping parsing separate from scoring means the same reducer works whether the confidence came from TSV, hOCR, or the ALTO XML that some access systems prefer.

from __future__ import annotations

import csv
import io
import logging
from dataclasses import dataclass

logger = logging.getLogger("archival.ocr.confidence")


@dataclass(frozen=True)
class Word:
    """A single recognized token and its Tesseract confidence (0–100)."""

    text: str
    conf: float


def parse_tsv_words(tsv_text: str) -> list[Word]:
    """Extract scored word tokens from Tesseract TSV output.

    Structural rows (page/block/paragraph/line) carry conf == -1 and no text;
    they are discarded so they cannot dilute the confidence distribution.
    """
    reader = csv.DictReader(io.StringIO(tsv_text), delimiter="\t")
    words: list[Word] = []
    for row in reader:
        text = (row.get("text") or "").strip()
        raw_conf = row.get("conf", "-1")
        try:
            conf = float(raw_conf)
        except ValueError:
            continue
        if not text or conf < 0:
            continue
        words.append(Word(text=text, conf=conf))
    logger.info("tsv_parsed", extra={"word_count": len(words)})
    return words

With clean tokens in hand, the reducer computes the three metrics the gate needs. Mean confidence catches the uniform depression of a low-DPI or wrong-model page. The low-confidence word ratio — the fraction of words scoring below a per-word floor — catches the banded collapse of skew and the ghost-character spike of bleed-through that a healthy mean can hide. Page coverage — the count of recognized words — catches the near-blank page whose handful of tokens happen to score high and would otherwise sail through on mean alone.

from statistics import fmean


@dataclass(frozen=True)
class PageMetrics:
    """Reduced confidence metrics for a single page."""

    word_count: int
    mean_conf: float
    low_conf_ratio: float


def compute_page_metrics(words: list[Word], low_word_floor: float = 60.0) -> PageMetrics:
    """Reduce a page's word stream to mean confidence and low-confidence ratio."""
    if not words:
        # A page with no recognized words is a coverage failure, not a high score.
        return PageMetrics(word_count=0, mean_conf=0.0, low_conf_ratio=1.0)

    confs = [w.conf for w in words]
    low_count = sum(1 for c in confs if c < low_word_floor)
    metrics = PageMetrics(
        word_count=len(words),
        mean_conf=round(fmean(confs), 2),
        low_conf_ratio=round(low_count / len(words), 4),
    )
    logger.info(
        "page_metrics_computed",
        extra={
            "word_count": metrics.word_count,
            "mean_conf": metrics.mean_conf,
            "low_conf_ratio": metrics.low_conf_ratio,
        },
    )
    return metrics

The gate itself is pure policy: a threshold object holds the promotion criteria, and the decision function returns a structured verdict with the reason for any rejection. Returning why a page failed — not just that it did — is what lets the routing layer send a skew failure to deskew-and-rescan and a degraded-original failure to manual transcription without re-deriving the diagnosis.

from dataclasses import asdict


@dataclass(frozen=True)
class GatePolicy:
    """Promotion thresholds. Tune per collection; historical material runs lower."""

    min_mean_conf: float = 80.0
    max_low_conf_ratio: float = 0.15
    min_word_count: int = 10


@dataclass(frozen=True)
class GateDecision:
    """The verdict for one page, carried into the preservation event."""

    promoted: bool
    reason: str
    metrics: PageMetrics


def evaluate_gate(metrics: PageMetrics, policy: GatePolicy) -> GateDecision:
    """Return a pass/fail promotion decision with a machine-readable reason."""
    failures: list[str] = []
    if metrics.word_count < policy.min_word_count:
        failures.append("insufficient_coverage")
    if metrics.mean_conf < policy.min_mean_conf:
        failures.append("mean_below_threshold")
    if metrics.low_conf_ratio > policy.max_low_conf_ratio:
        failures.append("low_conf_ratio_exceeded")

    decision = GateDecision(
        promoted=not failures,
        reason="pass" if not failures else ";".join(failures),
        metrics=metrics,
    )
    logger.info(
        "gate_evaluated",
        extra={"promoted": decision.promoted, "reason": decision.reason,
               **asdict(metrics)},
    )
    return decision

Confidence expresses a quality budget for the whole batch, not just a page. Averaging page means hides a few catastrophic pages inside a strong run, so the batch view weights by page and also reports the worst page and the count that failed the gate. Formally, weighting each page $p$ by its word count $w_p$ gives a batch mean that reflects text volume rather than page count:

$$\bar{c}{\text{batch}} = \frac{\sum w_p , \bar{c}p}{\sum w_p}, \qquad r_{\text{fail}} = \frac{\lvert { p : \text{gate}(p) = \text{fail} } \rvert}{N_{\text{pages}}}$$

def summarize_batch(decisions: list[GateDecision]) -> dict[str, float | int]:
    """Aggregate page decisions into a batch-level promotion summary."""
    total_words = sum(d.metrics.word_count for d in decisions)
    weighted = (
        sum(d.metrics.mean_conf * d.metrics.word_count for d in decisions) / total_words
        if total_words
        else 0.0
    )
    failed = [d for d in decisions if not d.promoted]
    summary = {
        "pages": len(decisions),
        "failed_pages": len(failed),
        "fail_ratio": round(len(failed) / len(decisions), 4) if decisions else 1.0,
        "batch_weighted_mean_conf": round(weighted, 2),
        "worst_page_mean_conf": round(min((d.metrics.mean_conf for d in decisions), default=0.0), 2),
    }
    logger.info("batch_summarized", extra=summary)
    return summary

Recognition is embarrassingly parallel, so in production the per-page reduction runs across the worker pools described in Parallelizing Tesseract OCR Across Worker Pools, and only the small GateDecision objects — not the full TSV — are gathered back for the batch summary and the promotion call.

# Validation and Verification

A gate is only trustworthy if its decisions are reproducible and recorded. Confirm the fix with the same rigor applied to fixity, not by eyeballing a sample:

• Re-run determinism. The reducer is a pure function of the TSV, so recomputing metrics from the archived TSV must reproduce the recorded mean_conf and low_conf_ratio byte-for-byte. Persist the TSV alongside the derivative; it is the evidence that justifies the promotion decision.
• Emit the preservation event before sealing. Every gate decision — pass and fail — records an OCR-quality event so the AIP carries proof that the text layer was measured, not assumed. Model it as a PREMIS validation event with the metrics in eventDetail, folded into provenance through PREMIS Metadata Mapping.

def build_ocr_quality_event(object_id: str, decision: GateDecision,
                            policy: GatePolicy) -> dict:
    """Construct a PREMIS-style validation event recording the gate outcome."""
    outcome = "pass" if decision.promoted else "fail"
    event = {
        "eventType": "validation",
        "eventDetail": "OCR confidence gate before AIP promotion",
        "linkingObjectId": object_id,
        "eventOutcome": outcome,
        "eventOutcomeDetail": {
            "reason": decision.reason,
            "mean_conf": decision.metrics.mean_conf,
            "low_conf_ratio": decision.metrics.low_conf_ratio,
            "word_count": decision.metrics.word_count,
            "policy": asdict(policy),
        },
    }
    logger.info("ocr_quality_event", extra={"object_id": object_id, "outcome": outcome})
    return event

• Gate the seal, not the log. The promotion step must consume decision.promoted as a hard precondition. A failed page cannot be written into the AIP under any code path; it is held in quarantine until a fresh capture passes. This is the same pre-seal discipline enforced end-to-end in End-to-End SIP-to-AIP Promotion, where OCR quality is one of several gates a package clears before archival storage.

# Edge Cases and Gotchas

The signature in the confidence distribution usually names the cause. This routing table turns each signature into an action rather than a generic re-scan:

Beyond the distribution signatures, four archival-specific traps recur:

• Confidence is not accuracy. Tesseract’s conf is the engine’s self-assessment, and it is systematically overconfident on a wrong language model — the fluent-garbage case. Treat a passing gate as necessary, not sufficient, for that material, and keep a human sampling loop for scripts outside the model’s training. The drift patterns behind this are covered in Handling OCR Drift in Historical Document Processing.
• Structural rows poison the mean. TSV rows for page, block, paragraph, and line levels carry conf == -1. Averaging them in drags every score toward zero and fails healthy pages; the parser above discards them, and any custom reducer must do the same.
• Whitespace-only tokens inflate coverage. Tesseract emits tokens whose text is a lone space or punctuation mark with a nominal confidence. Stripping and length-checking text before counting keeps word_count honest so a near-blank page cannot clear the coverage floor on formatting artifacts.
• One policy does not fit every collection. A clean twentieth-century typescript run justifies min_mean_conf = 90; a nineteenth-century newspaper morgue may top out near 75 even after perfect capture. Bind the GatePolicy to the collection or material type, version it alongside the batch profile, and record the applied policy in the event so a future auditor knows which bar the page cleared.

Tune thresholds against Tesseract’s own confidence and TSV output documentation and cross-check the resulting event structure against the LoC PREMIS Data Dictionary so the recorded outcome is interoperable with the rest of the preservation record.

# Frequently Asked Questions

# What confidence threshold should gate AIP promotion?

There is no universal number, because Tesseract’s conf is calibrated to image and script quality, not to a fixed accuracy target. Start from a mean-confidence floor near 80 for clean modern print and a low-confidence-word-ratio ceiling near 0.15, then adjust per collection: historical newspapers and handwritten material legitimately run 10–15 points lower even after ideal capture. The durable rule is to gate on two metrics — mean and the low-confidence ratio — because bleed-through and phantom characters can hold a healthy mean while the ratio spikes, and to version the policy so every promotion records which bar it cleared.

# Why gate on both mean confidence and the low-confidence word ratio?

Because each metric is blind to a different failure. Mean confidence catches the uniform depression of a low-DPI scan or a wrong language model, but it averages away a dense band of unreadable words inside an otherwise strong page — exactly the banded collapse that uncorrected skew and bleed-through produce. The low-confidence word ratio measures that tail directly. A page must satisfy both, plus a minimum word count, so that a near-blank page cannot pass on a handful of high-scoring tokens.

# What happens to a page that fails the gate?

It is quarantined — held out of the AIP entirely — and routed by the reason recorded in its decision. A resolution or skew failure goes back for re-scan or deskew and is re-OCR’d; a wrong-model failure is re-run with the correct language and needs no new capture; a degraded-original failure is routed to manual QA transcription because no engine parameter recovers information the paper no longer carries. In every case an OCR-quality preservation event is recorded first, so the archive has proof the text layer was measured before promotion was refused.

# Does a passing confidence gate guarantee accurate text?

No. Confidence is the engine’s self-assessment, and it is systematically overconfident when the wrong language model maps real glyphs onto plausible-but-wrong characters, producing fluent output with mid-range scores. A passing gate is a necessary floor that keeps obviously broken text layers out of the archive; it is not a substitute for a human sampling loop on scripts outside the model’s training. Treat the gate as the automated backstop and character-accuracy sampling as the audit above it.

# Related

• OCR Processing Pipelines — the parent stage: recognition architecture, engine configuration, and where confidence gating sits before promotion.
• Parallelizing Tesseract OCR Across Worker Pools — how the per-page reduction fans out across workers so only small decision objects return for the batch summary.
• Handling OCR Drift in Historical Document Processing — why self-reported confidence misleads on historical scripts and how drift shows up in the distribution.