ExifTool vs python-xmp-toolkit for Metadata Extraction

Every preservation master carries a payload of embedded metadata — capture timestamps, scanner make and model, ICC colour intent, IPTC rights, and the XMP packet that ties a derivative back to its source — and the tool you pick to read it decides how much of that survives into the archival record. This evaluation sits inside the Metadata Extraction Workflows stage of the Automated Ingestion & Batch Scanning Workflows pipeline, and it answers one recurring build decision: do you shell out to ExifTool for its exhaustive tag coverage, or bind directly to python-xmp-toolkit (Exempi’s C library) for tight, in-process XMP round-trips? The two are not interchangeable. ExifTool reads almost everything and writes it back conservatively; python-xmp-toolkit reads and rewrites XMP with library-grade precision but is blind to anything outside the XMP model. Choosing wrongly means either losing maker-note provenance you needed or bolting a 30 000-line Perl subprocess onto a job that only ever touches a Dublin Core packet.

# What Each Tool Actually Sees

The first and largest difference is namespace coverage, and it is not a matter of degree — the two tools model different universes.

ExifTool parses hundreds of metadata groups: EXIF/TIFF IFDs, GPS, IPTC IIM, XMP (every registered schema plus unknown namespaces preserved verbatim), ICC profile headers, JFIF, Photoshop IRBs, and — critically for provenance — manufacturer maker notes for Nikon, Canon, Epson, and the flatbed scanner firmware that stamps capture settings there. When a preservation master records its true optical resolution or a firmware build string in a proprietary maker note, ExifTool is frequently the only reader that surfaces it. It runs as an external process (Perl), emits structured JSON with -json, and normalises everything into flat Group:Tag keys.

python-xmp-toolkit is narrower by design. It wraps Exempi, the C port of Adobe’s XMP Toolkit SDK, and it sees exactly one thing: the XMP packet. It has no concept of an EXIF IFD or an IPTC IIM record unless that data has been reconciled into XMP. What it gives up in breadth it repays in fidelity — it parses the RDF/XML graph properly, understands qualifiers, language alternatives (rdf:Alt), ordered and unordered arrays (rdf:Seq/rdf:Bag), and struct nesting, and it can serialise the packet back out byte-for-byte within the XMP data model. For a workflow whose job is to read, amend, and re-embed an XMP packet — the work described in extracting embedded XMP metadata from TIFF files — that structural correctness matters more than raw tag count.

The Venn view below is the fastest way to internalise the split: a large ExifTool circle covering EXIF, IPTC, ICC and maker notes; a smaller, precise XMP circle where the tools overlap; and the decision rule that follows from where your required fields fall.

Coverage overlap drives the choice: ExifTool for anything outside the XMP packet, python-xmp-toolkit for precise in-process reads and re-embeds of XMP itself.

# Reading a Master with ExifTool via Subprocess

ExifTool exposes no native Python binding; you invoke the binary and parse its JSON. The -json flag emits an array of objects keyed by Group:Tag, -G prefixes each key with its group so EXIF:DateTimeOriginal never collides with XMP:CreateDate, and -n disables the human-readable print conversions so numeric values stay machine-parseable. Always pass arguments as a list (never shell=True) and never interpolate a filename into a shell string — an archival tree will eventually contain a path with a space, a quote, or a $.

import json
import logging
import subprocess
from pathlib import Path
from typing import Any

logger = logging.getLogger("archival.metadata.exiftool")


def extract_with_exiftool(path: Path, timeout_s: float = 60.0) -> dict[str, Any]:
    """Extract all embedded metadata from a preservation master via ExifTool.

    Returns a flat mapping of ``Group:Tag`` -> value. Raises on non-zero exit
    or malformed JSON so a failed read never masquerades as an empty record.
    """
    argv: list[str] = [
        "exiftool",
        "-json",   # structured output
        "-G",      # prefix every key with its group (EXIF:, XMP:, IPTC:, MakerNotes:)
        "-n",      # disable print conversion -> raw numeric values
        "-struct", # preserve XMP struct/array nesting instead of flattening
        str(path),
    ]
    proc = subprocess.run(
        argv,
        capture_output=True,
        text=True,
        timeout=timeout_s,
        check=False,  # inspect returncode explicitly for a precise log line
    )
    if proc.returncode != 0:
        logger.error(
            "exiftool_failed",
            extra={"file": str(path), "returncode": proc.returncode, "stderr": proc.stderr.strip()},
        )
        raise RuntimeError(f"ExifTool exited {proc.returncode} for {path}")

    records: list[dict[str, Any]] = json.loads(proc.stdout)
    fields: dict[str, Any] = records[0] if records else {}
    fields.pop("SourceFile", None)  # drop the echoed path; keep only metadata

    logger.info(
        "exiftool_extracted",
        extra={
            "file": str(path),
            "field_count": len(fields),
            "groups": sorted({k.split(":", 1)[0] for k in fields}),
        },
    )
    return fields

For a batch, the decisive tuning is to amortise interpreter startup. ExifTool’s Perl process takes tens to hundreds of milliseconds to launch, so per-file invocation dominates wall-clock time on large runs. Feeding many paths to one process with -@ argfile, or driving the persistent -stay_open True daemon, collapses thousands of process spawns into one. The helper below batches a whole directory through a single invocation.

def extract_batch_with_exiftool(paths: list[Path], timeout_s: float = 900.0) -> dict[str, dict[str, Any]]:
    """Extract metadata for many masters in one ExifTool process to amortise startup."""
    argv: list[str] = ["exiftool", "-json", "-G", "-n", "-struct", *[str(p) for p in paths]]
    proc = subprocess.run(argv, capture_output=True, text=True, timeout=timeout_s, check=True)
    records: list[dict[str, Any]] = json.loads(proc.stdout)

    by_file: dict[str, dict[str, Any]] = {}
    for rec in records:
        source = rec.pop("SourceFile")
        by_file[source] = rec
    logger.info("exiftool_batch_extracted", extra={"files": len(by_file), "requested": len(paths)})
    return by_file

# Reading the Same Master with python-xmp-toolkit

python-xmp-toolkit runs in-process against Exempi. XMPFiles opens the master and hands back an XMPMeta graph; you walk it by namespace URI and property name. There is no subprocess, no JSON round-trip, and no startup penalty — but you only ever see the XMP packet, so EXIF or maker-note fields that were never reconciled into XMP simply are not there.

import logging
from pathlib import Path
from typing import Any

from libxmp import XMPFiles, XMPMeta
from libxmp import consts as xmp_consts

logger = logging.getLogger("archival.metadata.xmp_toolkit")


def extract_with_xmp_toolkit(path: Path) -> dict[str, Any]:
    """Extract the embedded XMP packet in-process via Exempi (python-xmp-toolkit).

    Reads Dublin Core and XMP-basic properties from the RDF graph. Returns an
    empty mapping when the file carries no XMP packet, logging the distinction.
    """
    xmpfile = XMPFiles(file_path=str(path), open_onlyxmp=True)
    try:
        xmp: XMPMeta | None = xmpfile.get_xmp()
    finally:
        xmpfile.close_file()

    if xmp is None:
        logger.warning("xmp_absent", extra={"file": str(path)})
        return {}

    wanted: list[tuple[str, str, str]] = [
        (xmp_consts.XMP_NS_DC, "title", "dc:title"),
        (xmp_consts.XMP_NS_DC, "creator", "dc:creator"),
        (xmp_consts.XMP_NS_DC, "rights", "dc:rights"),
        (xmp_consts.XMP_NS_XMP, "CreateDate", "xmp:CreateDate"),
        (xmp_consts.XMP_NS_XMP, "CreatorTool", "xmp:CreatorTool"),
    ]
    fields: dict[str, Any] = {}
    for namespace, prop, label in wanted:
        if xmp.does_property_exist(namespace, prop):
            fields[label] = xmp.get_property(namespace, prop)

    logger.info(
        "xmp_extracted",
        extra={"file": str(path), "field_count": len(fields), "properties": sorted(fields)},
    )
    return fields

The value python-xmp-toolkit adds is not on the read path but on the write-back: because it owns a correct serialiser, put_property followed by put_xmp/can_put_xmp re-embeds the packet without corrupting the RDF or disturbing bytes outside the packet. That is exactly the property you want when normalising IPTC and EXIF timestamps across time zones and writing the corrected value back into the master rather than into a report.

# Comparison Matrix

# Verdict: When Each One Wins

Reach for ExifTool when breadth is the requirement. If you are surveying an unknown collection, harvesting technical characteristics for a JHOVE-style profile, or need a field that only exists in a maker note or a raw EXIF IFD — scanner firmware version, true optical DPI, GPS, ICC rendering intent — ExifTool is usually the only tool that surfaces it, and its -json output drops straight into a validation stage. Accept the operational cost: a Perl runtime on every worker, a subprocess boundary you must call safely, and startup latency you must amortise across a batch.

Reach for python-xmp-toolkit when the job is an embedded XMP round-trip. If the metadata that matters already lives in the XMP packet and you need to read it, amend it, and re-embed it inside the same process — with correct RDF semantics and no risk of a subprocess mangling the file — Exempi’s byte-accurate serialiser is the safer instrument, and it does it without spawning anything. Its narrowness is a feature here: it will not silently invent EXIF it cannot model.

Many production pipelines run both: ExifTool for the broad, one-shot technical survey at ingest, and python-xmp-toolkit for the targeted read-amend-write cycles that touch descriptive and rights metadata later. Whichever the master carries, the fields you extract should ultimately be mapped into a durable provenance record through PREMIS Metadata Mapping, so an extraction tool choice never becomes a chain-of-custody gap.

# Validation and Verification

Prove the extraction is faithful rather than trusting either tool’s exit code:

• Cross-check the overlap. For any master carrying an XMP packet, read xmp:CreateDate with both tools; the values must agree. A divergence means one path is reading a stale or duplicate packet and needs investigation before the value is trusted.
• Assert non-empty on required fields. Treat an empty python-xmp-toolkit result as a signal, not a success — it usually means the data lives in EXIF/IPTC and never reached XMP, so ExifTool is the correct reader for that field.
• Verify write-back byte safety. After a python-xmp-toolkit re-embed, re-read the file with ExifTool and confirm every non-XMP group (EXIF, ICC, maker notes) is byte-identical to the pre-edit read. This is the fixity discipline that keeps an edit auditable.
• Log the field inventory. Both extractors above emit a field_count and the groups/properties seen, so a batch produces a machine-parseable record of exactly what each tool recovered per object.

# Edge Cases and Gotchas

• Multi-page TIFFs. A bound-volume TIFF can hold a distinct XMP packet per IFD. ExifTool with -a -G4 surfaces per-page groups; python-xmp-toolkit’s XMPFiles reads the primary packet only, so per-page provenance needs an explicit page-by-page pass.
• Proprietary scanner extensions. Some flatbed firmware writes settings only into a private maker-note block. python-xmp-toolkit cannot see it; ExifTool decodes the documented ones and preserves the rest as an opaque binary tag.
• Legacy Latin-1 IPTC. IPTC IIM records predate mandatory UTF-8. ExifTool honours the CodedCharacterSet tag; if it is absent, force -charset iptc=Latin1 or non-ASCII rights strings will decode as mojibake.
• Exempi version skew. python-xmp-toolkit depends on the system libexempi; a wheel built against a newer Exempi than the host provides raises an opaque load error at import. Pin the system library in your deployment image, not just the Python package.
• Silent JSON truncation. ExifTool emits a partial record with a warning on a damaged file. Always inspect stderr and treat a non-empty warning stream as a reason to quarantine the object, not to record a thin metadata set as complete.

# Frequently Asked Questions

# Can python-xmp-toolkit read EXIF or IPTC fields directly?

No. It reads only the XMP packet through Exempi. EXIF and IPTC values appear to it exclusively when they have already been reconciled into XMP (for example an exif:DateTimeOriginal mirrored into the XMP exif namespace). If the field you need lives in a raw EXIF IFD, an IPTC IIM record, or a maker note, use ExifTool, which parses those groups natively.

# Is shelling out to ExifTool a security risk in a batch pipeline?

Only if you invoke it carelessly. Always pass arguments as an argv list — never shell=True and never interpolate filenames into a shell string — so a path containing spaces or shell metacharacters cannot inject a command. Beyond that, run the binary under a timeout, drop privileges on the worker, and treat a non-zero exit or non-empty stderr as a hard failure rather than an empty result.

# Which tool is faster for a large ingest batch?

python-xmp-toolkit has no per-file startup cost because it runs in-process, so for XMP-only work on many files it wins on raw latency. ExifTool pays a Perl startup penalty per invocation, but that vanishes when you batch many paths into a single process with -@ argfile or drive the -stay_open daemon. Amortised that way, ExifTool’s throughput on broad extraction is entirely competitive, and it does far more work per file.

# How do I safely write a corrected value back into the master?

Prefer python-xmp-toolkit when the change is confined to the XMP packet: put_property then put_xmp re-embeds the RDF byte-accurately and leaves every other segment of the file untouched. If you must edit an EXIF or IPTC field, ExifTool with -overwrite_original (or its automatic _original backup) is the right tool. In both cases, re-read the file afterward and confirm the untouched groups are byte-identical before recording the edit as a preservation event.

# Related

• Metadata Extraction Workflows — the parent stage: where embedded-metadata reads fit in the ingest pipeline and how extracted fields feed validation.
• Extracting Embedded XMP Metadata from TIFF Files — the deep dive on the XMP packet that python-xmp-toolkit round-trips.
• Normalizing IPTC and EXIF Timestamps Across Time Zones — reconciling the timestamp fields these tools extract into a single canonical value.