Provenance: what to capture from day one

Provenance is the difference between “we think it works” and “we can prove what ran.”

The easiest time to add provenance is day one—before systems spread across repos, services, datasets, vendors, and environments.

What to capture (the short checklist)

Code + configuration

• Git commit SHA (or immutable artifact digest)
• Feature flags and runtime configuration
• Prompt templates and versions (if you use prompts)

Data

• Dataset identifiers and versions
• Sampling logic and filters used for evaluation
• Access and retention constraints (what must not be stored)

Model artifacts

• Model name and version (or weights hash)
• Inference parameters (temperature, max tokens, etc.)
• Tooling versions (tokenizers, runtimes)

Environment

• Deployment target (cloud region, hardware class)
• Container image digest
• Dependency lockfiles

Evidence outputs

• Evaluation reports (with inputs, outputs, and scoring method)
• Known failure cases and their status
• Review approvals (who signed off, when, and for what scope)

The “reproducible evidence” test

If someone asked you to reproduce a result in 90 days, could you answer:

1. What ran? (exact code + config + prompt)
2. On what data? (versioned datasets and filters)
3. With what model? (versioned model artifacts)
4. Where did it run? (environment and dependencies)
5. What happened? (outputs and scoring)

If any answer is “we think…”, your provenance is incomplete.

Keep it lightweight

Provenance doesn’t need to be heavyweight logging. It needs to be:

• Immutable (hashes, digests, versions)
• Linkable (every evaluation links to its inputs and artifacts)
• Minimal (store only what you’re allowed to store)

In practice, the goal is to make provenance boring: part of your release process, not a scramble during an incident or audit.