Why every extracted field needs a confidence score

Consider a finance team running 5,000 invoices through a document-extraction tool each month. Even at 99% accuracy, that’s 50 wrong invoice totals per month slipping into the ledger. Most companies don’t have a process to catch them — because the tool said “extracted: £4,217.50” with no indication it might be wrong.

This is the core problem with extraction tools that don’t surface confidence scores. The accuracy number (99%, 95%, whatever) hides a uncomfortable truth: accuracy is the mean across easy and hard fields. The simple fields — vendor name on a clean PDF — extract at near-perfect rates. The hard ones — totals on handwritten invoices, dates on faxed receipts — extract at 60-80%, maybe lower. You can’t see which is which without a per-field confidence number.

Why “just trust the extraction” breaks at scale

Every extraction system has a performance cliff. Clean documents, standard formats, printed text: accuracy is excellent. Handwritten text, poor photocopies, unusual layouts: accuracy drops sharply. But the model doesn’t tell you when it’s standing at that cliff.

Vendors publish average accuracy figures because they sound good. A 99% figure suggests you only need to review 1 in 100 extractions. But if your data actually breaks down as:

• Vendor name: 99.5% (easy field)
• Invoice total: 82% (hard field)
• Date: 78% (very hard field)

…then reviewing every 100th extraction is not enough. You’re missing 18 wrong totals and 22 wrong dates per 100 documents.

At scale, this becomes expensive. Wrong amounts in your accounting. Wrong dates triggering false payment reminders. Wrong party details causing misrouted funds. The downstream cost of a single wrong extraction often exceeds the cost of human review, which means you need to review everything anyway — defeating the whole point of automation.

What a confidence score actually represents

A well-calibrated confidence score says: “this prediction is correct 90% of the time when I say I’m 90% confident.” That’s not a guess. It’s a statistical claim that can be tested and measured.

Beaqua publishes calibration curves per field type — for vendor names, for amounts, for dates, for contract terms — so you know what each confidence band actually means. A 95% confidence score on a vendor name means something different from a 95% confidence score on a handwritten amount: the first is probably right; the second should go to human review.

This transforms extraction from a black box (“trust it or don’t”) into a tool with visible error rates. You can pick thresholds based on your actual risk tolerance and the downstream cost of errors.

The 3-tier review workflow

Once you have confidence scores, you can build a proportionate review process:

• High confidence (≥95%): Accept silently. These extractions are correct often enough that manual review costs more than occasional errors. For vendor names, this is fine. For amounts, decide based on your risk appetite.
• Medium (70-95%): Accept with batch sampling. Spot-check a small percentage to catch systematic errors without reviewing everything.
• Low (less than 70%): Route to human review. These fields are unreliable; human judgment is faster than hoping the model guessed right.

Each tier covers a different cost profile. Tier 1 is cheap but risky. Tier 3 is safe but labour-intensive. Tier 2 is the pragmatic middle. The confidence score lets you run all three simultaneously on different fields, instead of forcing a binary choice: review everything or review nothing.

For a 5,000-invoice month, this might mean: 3,500 invoices skip review entirely (vendor name, date high-confidence). 1,200 invoices get sampled review (amounts 80-94% confidence). 300 invoices go to the review queue (amounts under 80% confidence). Total: maybe 150-200 human minutes instead of 5,000.

What we got wrong (and changed)

Early Beaqua confidence scores weren’t well-calibrated. We shipped them anyway. Customers told us a 70% score “felt” higher than 70% actually was — because they were comparing it to the vendor’s average-accuracy number, not to the ground truth. A 70% confidence score means 1 in 3 extractions is wrong, which is clearly bad, but our training data was biased toward easy documents.

We retrained the confidence head on a broader distribution of real customer documents — messy PDFs, handwritten forms, low-quality scans. The new scores are slower to train but honestly calibrated. A 70% score now means 70%, measured against real customer documents, not pristine test sets.

Lessons learned: calibration is a feature, not a footnote. It deserves the same engineering effort as the extraction model itself. If you’re evaluating a document-extraction tool and the vendor can’t show you calibration curves per field type, ask why.

Next steps

If you’re handling documents at scale and you’re not seeing per-field confidence scores, you’re flying blind. Confidence is the difference between automation that’s reliable enough to trust and automation that just pushes errors into your downstream systems.

See how Beaqua’s review workflow handles low-confidence fields in the product documentation.