Resolving DROID Format-Identification Conflicts

A clean DROID run reports exactly one PRONOM PUID per file, but production digitization batches rarely stay clean: a single object comes back with two competing PUIDs, a .docx is flagged as a plain ZIP archive, a truncated master resolves only by its extension, and an unrecognized proprietary scanner file returns nothing at all. This page is the conflict-resolution reference for the Preservation Format Identification stage — it picks up where Configuring Format Identification Tools Like DROID leaves off. Configuration gets DROID running against a current signature file; this guide answers the harder question that follows: when DROID hands you an ambiguous answer, how do you deterministically choose one canonical PUID, or decide that no automated answer is trustworthy and route the object to a human. Getting this wrong quietly poisons the format profile of an entire collection, and every downstream migration and rendering decision inherits the error.

# Why DROID Returns Conflicting Identifications

DROID matches a file three ways — by internal byte signature, by container inspection, and by filename extension — and records which method fired in the METHOD column of its CSV export. Conflicts arise because these methods disagree, or because a single method legitimately matches more than one format. The recurring root causes are concrete:

• Overlapping or generic signatures. Some PRONOM signatures are deliberately broad. A file whose leading bytes match both a generic wrapper and a specific profile produces two rows with the same DROID ID but different PUIDs — for example a JPEG that also satisfies a generic Exif container pattern. FORMAT_COUNT reports the number of hits, so any value above 1 is a multiple-match conflict by definition.
• ZIP-based container formats. OOXML (.docx, .xlsx), OpenDocument (.odt), and EPUB are all ZIP archives internally. If DROID’s container signatures are stale or container scanning is disabled, the byte signature resolves only to the generic ZIP PUID x-fmt/263 while the extension points at the real format — a classic signature-vs-extension split.
• Extension-vs-signature mismatch. DROID sets EXTENSION_MISMATCH=TRUE when the winning signature’s expected extensions do not include the file’s actual extension. This is the fingerprint of a renamed file, a mis-exported derivative, or an outright wrong extension applied at capture time.
• Truncated or embedded files. A master cut short by a failed transfer, or a payload extracted from a larger container, may match only a partial signature or match at an unexpected byte offset, yielding a tentative extension-only identification instead of a definitive signature hit.
• Outdated signatures. A file in a format newer than your installed signature release cannot be matched by signature at all. Until you refresh the registry — see syncing local PRONOM format-registry snapshots — DROID falls back to extension guessing or returns an empty result.

Every one of these produces one of four situations in the CSV: many PUIDs, exactly one PUID with a mismatch flag, one tentative extension-only PUID, or zero PUIDs. The decision tree below routes each to a deterministic outcome.

# Deciding Among Candidate Identifications

The core of resolution is a single, auditable policy applied per file: count the distinct candidate PUIDs, apply a strict method precedence (signature and container outrank extension), prefer the narrowest match, and escalate anything the rules cannot settle to manual review with the full candidate set recorded. The diagram encodes exactly the control flow implemented in the Python that follows.

Each DROID result is routed by candidate count: none and unresolved ambiguity escalate to manual review with the candidate set recorded, while a single trusted signature or a unique narrowest winner resolves automatically.

# A Precedence Policy for Canonical PUID Selection

The policy is deliberately narrow and ordered so that two archivists running it on the same CSV always reach the same canonical PUID. Encode it as explicit rules rather than tool defaults:

1. Signature and container beat extension. DROID’s METHOD column is ranked Container and Signature above Extension. An extension-only match is tentative and never wins over a byte-level identification, because the bytes are the object and the filename is a label an operator can change.
2. Narrowest PUID wins within a tier. When two signature-level PUIDs both match, prefer the more specific one. A versioned or profiled format (for example fmt/477, PDF/A-2b) is narrower than a generic parent (fmt/276, generic PDF), and the specific PUID is the correct canonical answer.
3. A mismatch flag on a signature ID is a warning, not a veto. If a definitive signature identifies the format but EXTENSION_MISMATCH=TRUE, trust the signature and record the mismatch as a note; the file was misnamed, not misidentified.
4. Cross-check anything the rules leave ambiguous. Two equally specific PUIDs, or a signature that resolves only to a generic ZIP for a known office document, go to a second tool — Siegfried, which reads the same PRONOM registry — and to container inspection before any human sees them.
5. Record the full candidate set on escalation. Manual review is only auditable if the reviewer receives every PUID DROID proposed, the method that produced each, and the mismatch state. Never discard the losing candidates.

# Parsing and Resolving a DROID Result Set

The module below reads a DROID CSV export, groups rows by file, applies the precedence policy, and emits a structured decision per file. It is deterministic and side-effect free: it never mutates files, only classifies them, so it is safe to run repeatedly over the same batch. Every candidate set and every decision is logged with extra= so the resolution is fully reconstructable from the log stream.

from __future__ import annotations

import csv
import logging
from collections import defaultdict
from dataclasses import dataclass, field
from enum import IntEnum
from typing import Iterable

logger = logging.getLogger("archival.format_id")

# DROID METHOD values ranked by trust: bytes beat filenames.
class MethodRank(IntEnum):
    NONE = 0
    EXTENSION = 1
    CONTAINER = 2
    SIGNATURE = 3


_METHOD_RANK = {
    "signature": MethodRank.SIGNATURE,
    "container": MethodRank.CONTAINER,
    "extension": MethodRank.EXTENSION,
    "": MethodRank.NONE,
}

_GENERIC_PUIDS = {"x-fmt/263", "fmt/276"}  # generic ZIP, generic PDF


@dataclass(frozen=True)
class Candidate:
    """One DROID identification hit for a single file."""
    puid: str
    method: MethodRank
    format_name: str
    format_version: str
    extension_mismatch: bool

    @property
    def specificity(self) -> tuple[int, int, int]:
        """Higher tuples are narrower: prefer versioned, non-generic PUIDs."""
        return (
            int(bool(self.format_version)),
            int(self.puid not in _GENERIC_PUIDS),
            len(self.puid),
        )


@dataclass
class Decision:
    """The resolved outcome for one file."""
    file_id: str
    name: str
    status: str                       # resolved | tentative | conflict | empty
    canonical_puid: str | None
    reason: str
    candidates: list[Candidate] = field(default_factory=list)


def _parse_bool(raw: str) -> bool:
    return raw.strip().lower() == "true"


def load_candidates(csv_path: str) -> dict[str, tuple[str, list[Candidate]]]:
    """Group DROID CSV rows into (file_name, candidates) keyed by DROID ID."""
    grouped: dict[str, tuple[str, list[Candidate]]] = {}
    rows_by_id: dict[str, list[dict[str, str]]] = defaultdict(list)
    with open(csv_path, newline="", encoding="utf-8") as handle:
        for row in csv.DictReader(handle):
            rows_by_id[row["ID"]].append(row)

    for file_id, rows in rows_by_id.items():
        name = rows[0].get("NAME", "")
        candidates: list[Candidate] = []
        for row in rows:
            puid = row.get("PUID", "").strip()
            if not puid:
                continue  # a row with no PUID contributes no candidate
            method = _METHOD_RANK.get(row.get("METHOD", "").strip().lower(),
                                      MethodRank.NONE)
            candidates.append(
                Candidate(
                    puid=puid,
                    method=method,
                    format_name=row.get("FORMAT_NAME", ""),
                    format_version=row.get("FORMAT_VERSION", "").strip(),
                    extension_mismatch=_parse_bool(row.get("EXTENSION_MISMATCH", "")),
                )
            )
        grouped[file_id] = (name, candidates)
    return grouped


def resolve(file_id: str, name: str, candidates: list[Candidate]) -> Decision:
    """Apply the precedence policy to a single file's candidate set."""
    logger.info(
        "candidates_loaded",
        extra={"file_id": file_id, "name": name,
               "candidate_puids": [c.puid for c in candidates],
               "count": len(candidates)},
    )

    if not candidates:
        return _record(Decision(file_id, name, "empty", None,
                                "no signature or extension match", candidates))

    # Rule 1: keep only the highest-trust method tier present.
    top_rank = max(c.method for c in candidates)
    tier = [c for c in candidates if c.method == top_rank]
    distinct_puids = {c.puid for c in tier}

    if top_rank == MethodRank.EXTENSION:
        # Only a filename-based guess survived — never authoritative.
        best = max(tier, key=lambda c: c.specificity)
        return _record(Decision(file_id, name, "tentative", best.puid,
                                "extension-only match; cross-check required",
                                candidates))

    if len(distinct_puids) == 1:
        winner = max(tier, key=lambda c: c.specificity)
        reason = ("signature match with extension mismatch (name suspect)"
                  if winner.extension_mismatch else "unambiguous signature match")
        return _record(Decision(file_id, name, "resolved", winner.puid,
                                reason, candidates))

    # Rule 2: multiple PUIDs in the trusted tier — try the narrowest.
    ranked = sorted(tier, key=lambda c: c.specificity, reverse=True)
    if ranked[0].specificity > ranked[1].specificity:
        return _record(Decision(file_id, name, "resolved", ranked[0].puid,
                                "narrowest PUID selected by precedence", candidates))

    # Rule 4/5: equally specific rivals — escalate with the full set intact.
    return _record(Decision(file_id, name, "conflict", None,
                            f"ambiguous PUIDs {sorted(distinct_puids)}; "
                            "cross-check Siegfried and inspect container", candidates))


def _record(decision: Decision) -> Decision:
    logger.info(
        "identification_decision",
        extra={"file_id": decision.file_id, "status": decision.status,
               "canonical_puid": decision.canonical_puid, "reason": decision.reason},
    )
    return decision


def resolve_batch(csv_path: str) -> list[Decision]:
    """Resolve every file in a DROID export, returning one Decision each."""
    decisions = [resolve(fid, name, cands)
                 for fid, (name, cands) in load_candidates(csv_path).items()]
    needs_review = [d for d in decisions if d.status in {"conflict", "empty", "tentative"}]
    logger.info(
        "batch_resolved",
        extra={"total": len(decisions), "manual_review": len(needs_review),
               "resolved": len(decisions) - len(needs_review)},
    )
    return decisions


def iter_review_queue(decisions: Iterable[Decision]) -> Iterable[Decision]:
    """Yield only the objects a human must adjudicate."""
    return (d for d in decisions if d.canonical_puid is None or d.status == "tentative")


if __name__ == "__main__":
    logging.basicConfig(level=logging.INFO)
    for outcome in resolve_batch("droid_export.csv"):
        print(outcome.status, outcome.canonical_puid, outcome.name)

The function returns a conflict with canonical_puid=None whenever two equally specific signature PUIDs survive, so no arbitrary tie-break silently commits a guess to the archival record. Those objects, plus every empty and tentative result, flow to the review queue.

# Validation and Verification

Confirm the resolution before trusting it, using the same evidence an auditor would demand:

• Cross-check with Siegfried. Run a second identifier over the disputed files. Because Siegfried reads the same PRONOM registry, agreement on the PUID is strong corroboration; disagreement means both tools are guessing and the object belongs in manual review regardless of what the precedence policy chose.

# Cross-check the disputed set with Siegfried and emit machine-readable output.
sf -csv -sig latest.sig ./review_queue/ > siegfried_crosscheck.csv

• Inspect ZIP-based containers directly. For a generic x-fmt/263 on a file that should be an office document, list the archive members: an OOXML package contains [Content_Types].xml and a word/ or xl/ directory, an ODT contains a mimetype member whose bytes name the format. The presence of that member confirms the specific format the byte signature failed to reach.
• Re-check magic bytes at the declared offset. For truncated or embedded files, read the first bytes and compare them against the PRONOM signature’s expected byte sequence and offset. A match at offset zero that fails deeper in the file confirms truncation rather than misidentification.
• Assert the registry is current. A surprising cluster of empty or extension-only results usually means the signature file predates the format. Re-run after refreshing snapshots, as covered under Format Registry Integration, before concluding a file is genuinely unidentifiable.
• Record the decision as a preservation event. Each resolved PUID and each escalation should emit a format identification event capturing the tool, the signature version, the chosen PUID, and the losing candidates — the provenance that makes the identification defensible under ISO 16363.

# Edge Cases and Gotchas

Two archival-specific traps deserve emphasis. First, multi-page and bound-volume TIFFs occasionally match both a baseline and a big-TIFF profile; the narrowest-PUID rule handles this correctly only if your signature snapshot actually contains the specific profile, so a mysterious tie is often a stale-registry symptom in disguise. Second, do not auto-correct extensions during identification — rename operations belong to a separate, logged normalization step, because silently changing a filename while resolving its format destroys the mismatch evidence a reviewer needs and can mask a deeper capture-pipeline defect.

# Frequently Asked Questions

# Should the file extension or the byte signature win when they disagree?

The byte signature, without exception. DROID’s METHOD column ranks a signature or container match above an extension match because the bytes are the actual object while the extension is an operator-supplied label that is trivially wrong after a bad export or manual rename. When a definitive signature identifies the format but EXTENSION_MISMATCH=TRUE, record the mismatch as a note and keep the signature’s PUID as canonical. The only case where an extension-based identification stands alone is when no signature matched at all, and that result is tentative by definition and must be cross-checked before you rely on it.

# Why does DROID report a .docx file as a generic ZIP archive?

OOXML, OpenDocument, and EPUB files are ZIP archives internally, so their outermost bytes match the generic ZIP signature x-fmt/263. DROID reaches the specific format only through container identification, which reads members inside the archive such as [Content_Types].xml. If container scanning is disabled or your container signatures are out of date, DROID stops at the generic ZIP PUID. Enable container identification and refresh the signature file; to confirm manually, list the archive members and look for the format-specific entries that distinguish an OOXML or ODT package from a plain ZIP.

# How do I choose between two PUIDs that both match by signature?

Apply the narrowest-match rule: prefer the more specific PUID over its generic parent. A versioned or profiled format identifier is narrower than a broad family identifier, and the specific one is the correct canonical answer. When the two candidates are equally specific — genuinely indistinguishable by the registry — the policy must not guess. Flag the object as a conflict, record both PUIDs and the method that produced each, cross-check with Siegfried, and route it to a human. An arbitrary tie-break committed to the archival record is worse than an honest escalation.

# What should happen to files DROID cannot identify at all?

An empty result is a routing decision, not a dead end. First rule out a stale registry, because a file in a format newer than your installed signatures cannot match by signature; refresh the PRONOM snapshot and re-run. If it is still unidentified, cross-check with Siegfried and inspect the leading bytes for a recognizable magic number. Only after those steps do you route the object to manual review, and you record the empty candidate set explicitly so the reviewer knows both tools failed rather than assuming the analysis was skipped.

# Related

• Configuring Format Identification Tools Like DROID — the setup this guide builds on: signature files, container scanning, and CSV export options.
• Syncing Local PRONOM Format-Registry Snapshots — keeping signatures current so stale-registry conflicts and empty results do not accumulate.
• Format Registry Integration — how PRONOM PUIDs feed normalization and migration-planning decisions downstream of identification.