Modeling Representation Information in AIP Packaging

A bitstream that survives a hundred years of storage is worthless if no future reader can turn its bytes back into a document. This is the problem OAIS Representation Information exists to solve, and it is the one most implementations quietly ignore. This walkthrough sits under the OAIS Reference Model Implementation area and picks up exactly where Setting Up OAIS SIP/AIP/DIS Workflows in Python stops: you have a validated Archival Information Package with a Content Data Object inside it, and now you must model the Representation Information that makes those bytes interpretable — and bind it into the package so it travels with the object rather than living in an archivist’s head. The failure this page prevents is the silent one: an AIP that passes every fixity check for decades and is then found, at migration time, to be an undecodable blob because the format specification, character encoding, or code book needed to parse it was never captured.

# Root-Cause Analysis of Un-Interpretable AIPs

OAIS (ISO 14721) defines an Information Object as a Data Object plus the Representation Information required to interpret it. Preserve only the Data Object and you have preserved bits, not information. The concrete ways this goes wrong recur across repositories:

1. Bits preserved without a format specification. The AIP stores a .tif bitstream but records no PRONOM PUID and no pointer to the TIFF 6.0 specification. A future agent must guess the byte layout — endianness, tag structure, strip offsets — from the extension alone. When the extension is wrong or proprietary, decoding is impossible.
2. Missing character encoding. A text or XML Content Data Object is stored with no declared encoding. Latin-1, UTF-8, and Windows-1252 share a byte range but disagree on every character above 0x7F, so a reader that guesses wrong silently corrupts the transcription of a manuscript.
3. Absent code books and code lists. A CSV of survey responses stores status=3 with no lookup table defining what 3 means. The Semantic Representation Information — the controlled vocabulary — was never captured, so the numbers outlive their meaning.
4. Broken Representation Networks. Structure Representation Information for a JPEG 2000 codestream depends on the ISO/IEC 15444-1 specification, which itself references the JPEG 2000 Part 1 markers document. If any link in that recursive chain is recorded as a bare label instead of a resolvable dependency, the network dangles and interpretation stalls at the missing node.
5. RepInfo stored but not bound to the object. The specifications exist somewhere on a shared drive, but the AIP’s METS or BagIt structure contains no link from the Content Data Object to them. Nothing in the package tells a future system which specification applies to which bitstream, so the association is lost the moment the person who knew it leaves.

The remedy is to model three layers explicitly — the Content Data Object, its Structure Representation Information (byte layout, format specification, PUID), and its Semantic Representation Information (encoding, code lists, data dictionaries) — then chain them into a Representation Network whose recursive dependencies terminate at a Designated Community’s knowledge base. The diagram below traces one such network from raw codestream to the assumed knowledge that needs no further explanation.

A Representation Network resolves recursively from the Content Data Object through structure and semantic layers until every chain terminates at the Designated Community’s assumed knowledge; a link that resolves to nothing (amber) is a dangling dependency the resolver must flag.

# Step-by-Step Resolution: Model the Layers, Then Walk the Network

# Model Representation Information as typed records

Represent each node in the network as a typed object carrying its category (structure, semantic, or the terminating “other”/knowledge-base sentinel), a resolvable identifier, and an explicit list of the identifiers it depends on. Binding a PUID here is what lets the same record reconcile against a format registry later, the concern owned by Format Registry Integration.

from __future__ import annotations

import logging
from enum import Enum

from pydantic import BaseModel, Field

logger = logging.getLogger("archival.repinfo")


class RepInfoCategory(str, Enum):
    """OAIS Representation Information categories plus a terminating sentinel."""

    STRUCTURE = "structure"
    SEMANTIC = "semantic"
    KNOWLEDGE_BASE = "knowledge_base"  # assumed knowledge; terminates the network


class RepresentationInformation(BaseModel):
    """One node in a Representation Network.

    A node interprets either the Content Data Object or another RepInfo node.
    ``depends_on`` names the identifiers this node itself requires; an empty
    list is only valid for the KNOWLEDGE_BASE sentinel.
    """

    rep_id: str = Field(..., description="Stable, resolvable identifier, e.g. 'rep-info-2'.")
    category: RepInfoCategory
    label: str
    puid: str | None = Field(default=None, description="PRONOM PUID when this node is a format spec.")
    encoding: str | None = Field(default=None, description="Character encoding for semantic text RepInfo.")
    locator: str | None = Field(default=None, description="Path or URI to the stored specification bytes.")
    depends_on: list[str] = Field(default_factory=list)


class ContentDataObject(BaseModel):
    """The preserved bitstream plus the RepInfo roots that interpret it."""

    object_id: str
    bitstream_path: str
    puid: str
    structure_rep: str  # rep_id of the root structure node
    semantic_rep: str   # rep_id of the root semantic node

# Walk the Representation Network and flag dangling dependencies

The resolver performs a depth-first traversal from the Content Data Object’s two roots, resolving every depends_on identifier against the registry of known nodes. A node whose dependency does not resolve is a dangling edge — the exact defect that renders an AIP un-interpretable years later — so it is logged at error level and collected rather than swallowed. Cycles are guarded with a visited set so a self-referential specification cannot spin forever.

class DanglingDependencyError(Exception):
    """Raised when a Representation Network fails to resolve to a knowledge base."""


class RepresentationNetworkResolver:
    """Resolves and validates a Representation Network for one Content Data Object."""

    def __init__(self, nodes: dict[str, RepresentationInformation]) -> None:
        self._nodes = nodes

    def resolve(self, cdo: ContentDataObject) -> list[str]:
        """Walk the network from both roots; return the resolution order.

        Raises DanglingDependencyError if any dependency is unresolved or if a
        chain never terminates at a KNOWLEDGE_BASE node.
        """
        visited: set[str] = set()
        order: list[str] = []
        dangling: list[str] = []
        terminated = False

        def walk(rep_id: str, parent: str) -> None:
            nonlocal terminated
            node = self._nodes.get(rep_id)
            if node is None:
                dangling.append(rep_id)
                logger.error(
                    "repinfo_dangling",
                    extra={"object_id": cdo.object_id, "missing": rep_id, "referenced_by": parent},
                )
                return
            if rep_id in visited:
                return
            visited.add(rep_id)
            if node.category is RepInfoCategory.KNOWLEDGE_BASE:
                terminated = True
            logger.info(
                "repinfo_resolved",
                extra={
                    "object_id": cdo.object_id,
                    "rep_id": node.rep_id,
                    "category": node.category.value,
                    "puid": node.puid,
                    "depends_on": node.depends_on,
                },
            )
            order.append(rep_id)
            for dep in node.depends_on:
                walk(dep, parent=rep_id)

        for root in (cdo.structure_rep, cdo.semantic_rep):
            walk(root, parent=cdo.object_id)

        if dangling:
            raise DanglingDependencyError(
                f"{cdo.object_id}: unresolved RepInfo {dangling}"
            )
        if not terminated:
            raise DanglingDependencyError(
                f"{cdo.object_id}: network never reaches a knowledge base"
            )
        return order

# Bind the resolved network into the AIP

A resolved network is only durable if it is serialized into the package. In a METS-based AIP, each Representation Information node becomes a <file> in a dedicated RepInfo file group, and the structMap links the Content Data Object’s <div> to those files so the association is machine-readable. The FILEID references make the Representation Network navigable directly from the object’s structure.

import xml.etree.ElementTree as ET

METS = "http://www.loc.gov/METS/"


def build_repinfo_structmap(
    cdo: ContentDataObject,
    order: list[str],
    nodes: dict[str, RepresentationInformation],
) -> ET.Element:
    """Emit a METS structMap div binding the CDO to its resolved RepInfo files."""
    ET.register_namespace("mets", METS)
    struct_map = ET.Element(f"{{{METS}}}structMap", {"TYPE": "logical"})
    obj_div = ET.SubElement(
        struct_map, f"{{{METS}}}div",
        {"TYPE": "content-data-object", "ID": cdo.object_id, "CONTENTIDS": cdo.puid},
    )
    ET.SubElement(obj_div, f"{{{METS}}}fptr", {"FILEID": f"file-{cdo.object_id}"})
    rep_div = ET.SubElement(
        obj_div, f"{{{METS}}}div", {"TYPE": "representation-network", "ID": f"repnet-{cdo.object_id}"},
    )
    for rep_id in order:
        node = nodes[rep_id]
        ET.SubElement(
            rep_div, f"{{{METS}}}fptr",
            {"FILEID": rep_id, "CONTENTIDS": node.puid or node.encoding or "assumed-knowledge"},
        )
        logger.info("repinfo_bound", extra={"object_id": cdo.object_id, "rep_id": rep_id})
    return struct_map

For a BagIt AIP, the same binding is expressed with tag files instead of XML: the Representation Information specifications live under data/repinfo/, and a repinfo-manifest.txt tag file plus bag-info.txt entries record each PUID and the depends_on edges, so bagit fixity covers the RepInfo bytes alongside the Content Data Object. Either way, the object and the information needed to read it are fixity-protected together and travel as one unit. The provenance of when each RepInfo node was captured is emitted as a PREMIS event, following PREMIS Metadata Mapping.

# Validation and Verification

Confirm the model actually protects interpretability rather than merely asserting it:

• Assert every chain terminates. Run the resolver over the packaged AIP and require it to return without raising; a DanglingDependencyError is a hard build failure, not a warning. A network that does not reach a KNOWLEDGE_BASE node is un-interpretable by definition.
• Reconcile PUIDs against the registry. Every puid on a structure node must resolve in your local PRONOM snapshot. An unknown PUID means the format specification is unbound — the same class of defect handled by Format Registry Integration.
• Round-trip the encoding. For semantic text RepInfo, decode a sample of the Content Data Object with the declared encoding and confirm no replacement characters (U+FFFD) appear; a guess that “looks fine” in UTF-8 but was authored in Windows-1252 fails here.
• Verify the binding survives extraction. Extract the METS structMap (or BagIt tag files) from the stored AIP and re-run the resolver against only what the package contains. If resolution depends on anything outside the bag, the network is not self-contained.

# Edge Cases and Gotchas

Three archival-specific traps deserve extra care. A multi-page TIFF carries one structure specification for the container but may embed several photometric interpretations across strips; model the photometric code as semantic RepInfo, not structure, or a bilevel scan will be rendered as if it were greyscale. Proprietary scanner extensions — vendor private TIFF tags — have no PRONOM PUID; capture the vendor’s tag documentation as its own RepInfo node and mark the dependency explicitly rather than letting it dangle. And legacy Latin-1 finding aids frequently arrive with a UTF-8 declaration that is simply wrong; trust the byte histogram, not the XML prolog, when you assign the encoding on the semantic node.

# Frequently Asked Questions

# What is the difference between Structure and Semantic Representation Information?

Structure Representation Information describes the form of the bits — the byte layout, the container format, the codestream markers — and is what lets a parser turn a bitstream into typed values such as pixels, integers, or characters. It is anchored to a format specification and, ideally, a PRONOM PUID. Semantic Representation Information describes what those values mean: the character encoding of text, the colour space of an image, or the code list that says status=3 means “withdrawn”. You need both. Structure alone parses bytes into numbers nobody can interpret; semantics alone has nothing to attach to. OAIS treats the pair, together with the Data Object, as the complete Information Object.

# When does a Representation Network stop expanding?

A Representation Network is recursive: structure RepInfo may depend on a format specification, which depends on a character set standard, and so on. It terminates when a node reaches the Designated Community’s knowledge base — the body of knowledge you are entitled to assume a future reader already has, such as the ability to read English or understand ASCII. That terminating node needs no further Representation Information. Defining the Designated Community is therefore an archival policy decision, not a technical one: a narrower community assumes less and forces you to capture more RepInfo explicitly. A network that never reaches such a node is, by OAIS’s own definition, incompletely specified.

# Should Representation Information live inside the AIP or in a shared registry?

Both, with a clear rule about which is authoritative. The AIP must contain — or contain a fixity-protected, resolvable reference to — every specification its Content Data Object depends on, so the package is self-describing when extracted in isolation. A shared Representation Information registry avoids storing the TIFF 6.0 specification in ten thousand bags, but the bag must still bind each node by a stable identifier and PUID, and your preservation policy must guarantee that registry outlives the AIP. If the registry can disappear while the AIP persists, embed the specifications in the payload instead.

# How do I bind Representation Information into a METS or BagIt AIP?

In METS, store each Representation Information node as a <file> in a dedicated file group and reference it from the Content Data Object’s structMap div via fptr FILEID links, carrying the PUID in CONTENTIDS. In BagIt, place the specifications under data/repinfo/ so the payload manifest’s checksums cover them, and record the PUIDs and dependency edges in a tag file and bag-info.txt. Either way the goal is identical: a machine can start at the object and follow explicit links to everything needed to interpret it, and fixity protects the Representation Information alongside the bytes it explains.

# Related

• OAIS Reference Model Implementation — the parent area: OAIS entities, package types, and where Representation Information fits in the Information Model.
• Setting Up OAIS SIP/AIP/DIS Workflows in Python — how the AIP that carries this Representation Network is assembled from a validated submission.
• Format Registry Integration — reconciling the PUIDs bound to structure RepInfo against a local PRONOM snapshot.