Validating Schema Compliance During Digital Ingest: Troubleshooting Edge Cases in Archival Workflows

Digital preservation pipelines require deterministic metadata and structural compliance before assets enter long-term storage. This page is the edge-case troubleshooting reference for the schema layer defined in Batch Validation Schemas: it takes the layered gate specified there and shows what to do when a package is nearly valid but fails for reasons that have nothing to do with malformed metadata — a superseded namespace URI, a Windows-1252 byte in an OCR text layer, or a manifest that reached the validator before capture finished writing. As institutional digitization scales to thousands of objects per day inside the broader Automated Ingestion & Batch Scanning Workflows pipeline, these failures stop being rare curiosities and become the dominant reason batches stall. The specific problem solved here is the gap between “the validator raised an exception” and “the package is genuinely non-compliant” — the ambiguity that causes engineers to reject perfectly recoverable content or, worse, to loosen the schema until it no longer protects the repository.

# Root-Cause Analysis of Ingest Validation Failures

Validation failures in archival workflows rarely stem from malformed XML alone. They typically originate upstream in device-generated manifests or downstream in automated enrichment stages. Three primary failure modes dominate production environments:

1. Namespace drift and schema version mismatch. METS, ALTO, and PREMIS packages frequently inherit outdated or conflicting namespace declarations from scanning hardware or legacy export tools. Because XSD validation is namespace-aware, an instance carrying a superseded namespace URI no longer binds to the expected element declarations, so lxml reports it as DocumentInvalid (unresolvable elements) rather than a structural content error — obscuring the true root cause. Resolving the correct target namespace is the same lookup problem handled by Format Registry Integration, where version-pinned identifiers are mapped to their current authoritative form.
2. Mixed character encodings. Legacy Dublin Core exports or OCR text layers often interleave UTF-8, ISO-8859-1, and Windows-1252 byte sequences. Standard validators fail immediately with UnicodeDecodeError, preventing any schema evaluation before a single element is inspected.
3. Asynchronous race conditions. When Scanner API Integration & Routing pushes a manifest to the validation queue while OCR Processing Pipelines simultaneously append structural metadata, an incomplete payload reaches the validator. The result is missing required elements or orphaned file references that look like schema violations but are really timing artifacts.

Debugging these failures requires inspecting the validator’s native error log rather than relying on generic traceback messages. For lxml, the schema.error_log object provides precise line/column coordinates and XPath locations. Mapping these coordinates back to the original ingest manifest enables targeted remediation rather than blanket rejection of an entire batch.

The decision tree below maps the dominant failure signatures to their remediation paths.

Each native error signature points to a specific remediation rather than a blanket rejection of the batch.

# Step 1: Pre-Validation Normalization

Before schema evaluation begins, every payload must pass through a deterministic normalization stage. This layer resolves encoding ambiguities and sanitizes structural anomalies that would otherwise trigger parser-level exceptions long before any schema assertion is reached.

import re
import logging
from pathlib import Path
import charset_normalizer

logger = logging.getLogger("ingest.normalize")


def normalize_xml_payload(file_path: Path, min_confidence: float = 0.75) -> bytes:
    """Detect encoding, decode, escape stray entities, and return clean UTF-8 bytes."""
    raw_bytes = file_path.read_bytes()

    match = charset_normalizer.from_bytes(raw_bytes).best()
    if match is None:
        logger.error("encoding_undetectable file=%s", file_path.name)
        raise ValueError(f"Could not detect encoding for {file_path.name}")

    encoding = match.encoding
    confidence = 1.0 - match.chaos
    if confidence < min_confidence:
        logger.error(
            "encoding_low_confidence file=%s encoding=%s confidence=%.2f",
            file_path.name, encoding, confidence,
        )
        raise ValueError(f"Low encoding confidence ({confidence:.2f}) for {file_path.name}")

    decoded = raw_bytes.decode(encoding, errors="replace").lstrip("")

    # Escape ampersands that are not already the start of a valid entity.
    decoded = re.sub(r"&(?!#?[a-zA-Z0-9]+;)", "&amp;", decoded)

    logger.info(
        "normalized file=%s source_encoding=%s confidence=%.2f",
        file_path.name, encoding, confidence,
    )
    return decoded.encode("utf-8")

This step eliminates XMLSyntaxError and UnicodeDecodeError before the schema validator is invoked, and it strips any leading byte-order mark. It also standardizes the byte stream, which matters when payloads are pulled across distributed storage tiers where chunked or compressed transfers are in use.

# Step 2: Dynamic Schema Registry and Namespace Remapping

Hardcoding a single XSD or JSON Schema into the pipeline creates immediate technical debt. Preservation standards evolve, and validation rules must adapt without code deployments. A dynamic schema registry — where rules are pulled from a versioned configuration service and namespace URIs are remapped to their current authoritative form — resolves namespace drift without loosening the schema itself.

import json
import logging
from lxml import etree
import jsonschema
from typing import Any

logger = logging.getLogger("ingest.validate")

# Legacy namespace URIs observed from older scanner exports mapped to current URIs.
NAMESPACE_REMAP: dict[str, str] = {
    "http://www.loc.gov/METS": "http://www.loc.gov/METS/",
    "info:lc/xmlns/premis-v2": "http://www.loc.gov/premis/v3",
}


def remap_namespaces(xml_bytes: bytes) -> bytes:
    """Rewrite superseded namespace URIs so instances bind to current declarations."""
    text = xml_bytes.decode("utf-8")
    for legacy, current in NAMESPACE_REMAP.items():
        if legacy in text:
            logger.info("namespace_remapped from=%s to=%s", legacy, current)
            text = text.replace(legacy, current)
    return text.encode("utf-8")


class SchemaValidator:
    def __init__(self, xsd_path: str, json_schema_path: str):
        self.xsd_schema = etree.XMLSchema(etree.parse(xsd_path))
        with open(json_schema_path, "r", encoding="utf-8") as fh:
            self.json_schema = json.load(fh)

    def validate_xml(self, xml_bytes: bytes) -> bool:
        doc = etree.fromstring(remap_namespaces(xml_bytes))
        if not self.xsd_schema.validate(doc):
            errors = [
                f"line {e.line}, col {e.column}, xpath {e.path}: {e.message}"
                for e in self.xsd_schema.error_log
            ]
            logger.warning("schema_invalid errors=%d", len(errors))
            raise etree.DocumentInvalid("\n".join(errors))
        return True

    def validate_json(self, payload: dict[str, Any]) -> bool:
        validator = jsonschema.Draft202012Validator(self.json_schema)
        errors = sorted(validator.iter_errors(payload), key=lambda e: list(e.path))
        if errors:
            for err in errors:
                logger.warning("json_violation path=%s message=%s", list(err.path), err.message)
            raise ValueError(f"{len(errors)} JSON Schema violation(s) at ingest")
        return True

By decoupling schema retrieval and namespace resolution from validation logic, teams can absorb schema migrations without redeploying the pipeline. The same architecture supports hybrid validation, where bagit-python verifies package integrity before lxml or jsonschema evaluates the metadata payloads, exactly as sequenced in the parent Batch Validation Schemas gate.

# Step 3: Async Execution With Error Isolation

Synchronous validation on monolithic batches causes memory exhaustion and timeout cascades, particularly when processing high-resolution TIFF derivatives or multi-gigabyte METS wrappers. Distributing validation across discrete worker processes isolates a single bad object from the rest of the batch. The re-queue path a race condition needs is owned by Error Handling & Retry Logic; the code below classifies each verdict so that transient failures route there instead of being marked permanently rejected.

import concurrent.futures
import logging
from dataclasses import dataclass

logger = logging.getLogger("ingest.batch")


@dataclass
class ValidationResult:
    file_id: str
    status: str          # "PASS" | "FAIL" | "RETRY"
    error_message: str | None


def _classify(exc: Exception) -> str:
    """Distinguish a genuine schema violation from a recoverable timing artifact."""
    msg = str(exc).lower()
    if "missing" in msg or "orphan" in msg or "not found" in msg:
        return "RETRY"          # likely an incomplete payload, re-queue with backoff
    return "FAIL"               # genuine schema violation, quarantine


def batch_validate(items: list[tuple[str, bytes]], max_workers: int = 8) -> list[ValidationResult]:
    validator = SchemaValidator("mets.xsd", "metadata_schema.json")
    results: list[ValidationResult] = []

    with concurrent.futures.ThreadPoolExecutor(max_workers=max_workers) as pool:
        futures = {pool.submit(validator.validate_xml, payload): fid for fid, payload in items}
        for future in concurrent.futures.as_completed(futures):
            fid = futures[future]
            try:
                future.result()
                results.append(ValidationResult(fid, "PASS", None))
                logger.info("validated file_id=%s verdict=PASS", fid)
            except Exception as exc:
                verdict = _classify(exc)
                results.append(ValidationResult(fid, verdict, str(exc)))
                logger.warning("validated file_id=%s verdict=%s", fid, verdict)
    return results

Combined with exponential backoff, this pattern lets transient network drops or a temporary schema-registry outage be retried before an object is marked permanently rejected — while a true schema_violation short-circuits straight to quarantine.

# Validation and Verification

A remediation is only trustworthy once you can prove the fix worked and record that proof. Three checks confirm a repaired package is genuinely compliant, and the third is what makes the verdict defensible under audit.

• Re-validate against the schema. Run the repaired bytes back through validate_xml/validate_json and assert an empty error_log. A normalization or remap that silences an exception without producing a clean schema pass has masked a defect, not resolved it.
• Assert fixity is unchanged. Normalization rewrites the byte stream, so recompute the SHA-256 of the content payload (not the sidecar) and confirm it still matches the manifest digest; an encoding fix must never alter a preservation master’s bytes.
• Emit a PREMIS validation event. Every verdict — pass or fail — is recorded as a preservation event, following PREMIS Metadata Mapping, so the audit trail captures the schema version in force and the remediation applied. This is the alignment point with the OAIS Reference Model Implementation: a validation event populates the Provenance information a package must carry to be a legitimate AIP.

import logging
from datetime import datetime, timezone

logger = logging.getLogger("ingest.premis")


def emit_validation_event(
    object_id: str,
    outcome: str,          # "pass" | "schema_violation" | "encoding_repaired" | ...
    schema_version: str,
    detail: str,
) -> dict[str, str]:
    """Build a PREMIS-shaped validation event for the audit ledger."""
    event = {
        "eventType": "validation",
        "eventDateTime": datetime.now(timezone.utc).isoformat(),
        "linkingObjectId": object_id,
        "eventOutcome": outcome,
        "eventOutcomeDetail": f"schema={schema_version}; {detail}",
    }
    logger.info(
        "premis_event object=%s outcome=%s schema=%s",
        object_id, outcome, schema_version,
    )
    return event

Inspecting these events after a remediation run is how a curator confirms a repaired batch was accepted for the right reason: a package that passed after an encoding_repaired event is compliant; a package that passed only after the schema was relaxed is a regression to hunt down.

# Edge Cases and Gotchas

• BOM-prefixed UTF-8 from Windows scanning stations. A leading  byte-order mark makes an otherwise valid document fail the XML declaration check. The lstrip("") in normalization removes it, but only after decoding — stripping the raw bytes before encoding detection can corrupt genuinely UTF-16 content.
• Namespace remap that is too greedy. A blind string replacement of a legacy namespace URI will also rewrite any occurrence inside descriptive text or a <premis:eventDetail> note. Anchor the remap to namespace-declaration positions, or validate that the replaced instance still round-trips, before trusting it on production masters.
• Multi-page TIFF manifests validated mid-write. A scanner packager that lists pages incrementally can present a manifest whose files[] array is complete but whose byte payloads are still flushing. Treat a RETRY verdict as gated on a capture-complete sentinel from Metadata Extraction Workflows, not on a fixed sleep.
• Legacy Latin-1 in a field the schema declares as free text. charset_normalizer may report high confidence for ISO-8859-1 content that is also valid UTF-8 for the ASCII range, silently passing a mojibake £ or é into a controlled-vocabulary field. Pin known-legacy producers to an explicit source encoding rather than relying on detection alone.

By treating validation as a stateless, idempotent service — normalize, remap, validate in isolation, then record the verdict — archival engineering teams resolve these edge cases deterministically instead of loosening the schema, protecting the integrity of long-term collections while keeping throughput high.

# Related

• Batch Validation Schemas — the parent gate: the four-layer validation stack and the manifest field contract this page troubleshoots.
• Error Handling & Retry Logic — where a RETRY verdict goes: backoff, quarantine, and re-ingest scheduling.
• PREMIS Metadata Mapping — how each validation verdict becomes a permanent, migration-surviving preservation event.