AI adoption and document workflows
AI document extraction: design the human review workflow first
AI can reduce manual entry from invoices, purchase orders and forms, but extraction is only one step. Before connecting results to an ERP or accounting system, define which documents are supported, how each field is tested, what a reviewer sees and which states a record passes through.
By SVS Digital · Published
Start here
Start with one bounded document type and a representative historical sample. Create verified reference values, measure results per field and derive review thresholds from your own errors and operating risks. Route uncertain or invalid results to a queue that shows the source document. Keep approved data in controlled write states, block duplicates and reconcile every attempted write. A confidence score is a routing signal, not proof that a value is correct.
1. Bound the document and decision scope
Choose one process first, such as supplier invoices received as PDFs by one finance team. Record document types, languages, page limits, image quality, layouts, channels and required fields. Keep invoices, purchase orders and general forms separate unless the same controls genuinely apply.
For each field, name its destination and the consequence of a wrong, missing or duplicate value. Define the business owner, exception approver and cases that stay manual. NIST recommends documenting purpose, context, users, limitations and human-oversight responsibilities before deployment.
- Name the document class, source channel and owner.
- List required, optional and prohibited fields.
- Define unsupported documents and their manual route.
2. Build a representative reference sample
Use historical documents that match real work: common suppliers, rare layouts, scans, rotated pages, multi-page tables, credit notes and poor images. Reserve some suppliers or periods for the final test so configuration work does not influence acceptance results.
Create a reference record for every test document. A trained operator should verify the source value, normalized value, location and any legitimate ambiguity. Resolve labelling disagreements before scoring. Google evaluates extraction by comparing predicted entities with test-document annotations and reports precision, recall and F1 per label as well as across labels.
- Sample the real mix of layouts and input quality.
- Include known exceptions, not only clean documents.
- Version documents, labels and normalization rules together.
3. Set acceptance rules per field
Measure precision, recall and F1 by field, not only as one document-wide average. Precision shows the share of predictions that match the annotations; recall shows the share of annotated values that are found. The useful trade-off depends on the field and the harm caused by a false acceptance or unnecessary review.
Derive confidence thresholds from representative documents and error costs. Test several cut-offs, inspect false accepts and missed values, then set a rule for each field or risk tier. Do not copy a vendor threshold. Google describes confidence as the model's association between an entity and its predicted value, while Microsoft notes that not every document field returns a confidence score. If a field has no usable score, require another validation or human review.
- Accept only when confidence and validation rules pass.
- Review when confidence is low, missing or untested.
- Use stricter treatment for high-impact fields.
4. Give reviewers the evidence they need
Show the source page, highlighted region, extracted value, normalized value, confidence when available and reason for review. Include nearby context such as currency, line description or supplier. Let reviewers confirm, correct, mark missing, reject or escalate.
Prioritize work by business risk and age, not confidence alone. Preserve the original prediction, correction, reviewer, timestamp and reason code. Audit both accepted and corrected items, and let reviewers report new layouts or recurring failures. This reduces rubber-stamping and creates evidence for improving the workflow.
- Keep the source image and field in one view.
- Explain why the item entered review.
- Define escalation and queue-time targets.
5. Validate, deduplicate and control writes
Run deterministic checks after extraction and correction: date and currency formats, required fields, supplier matching, tax calculations, line-total reconciliation and purchase-order status. Keep raw text beside normalized and approved values. A high confidence score must not override a failed business rule.
Use both a file fingerprint and a business duplicate key, such as supplier, invoice number and date. Check before approval and again before writing. Route suspected duplicates for resolution rather than silently dropping or posting them.
Use explicit states such as received, extracted, needs review, approved, rejected, write pending, written, write failed and reconciled. Prevent direct writes from extraction. Start in shadow mode or with drafts. For live writes, use an idempotency key where supported, capture the destination response and record identifier, and route failures to controlled recovery. Keep higher-risk posting manual until a separate acceptance case supports changing that boundary.
- Validate syntax, cross-field relationships and master data.
- Make retries idempotent and preserve the audit trail.
- Restrict who or what can move each record state.
6. Monitor quality, reconcile outcomes and keep a stop rule
Track volume, extraction failures, review rate, queue age, corrections, duplicate flags, write failures and reconciliation differences. Recheck field-level precision and recall on an audited sample that includes auto-accepted documents. A lower review rate is not success if false accepts rise.
Reconcile source documents, approvals and destination records by count and value. Break results down by supplier, layout, channel and model version. Pause auto-acceptance when a critical field misses its target, a new layout causes errors or destination reconciliation fails.
No extraction model has one accuracy figure that applies to every document, field and operating context. Results can change with scans, handwriting, unseen suppliers, layout changes, tables, languages and ambiguous values. Confidence helps route work but does not establish document authenticity, accounting treatment, tax correctness or legal compliance. Begin with a limited document group and reversible write scope, and expand only when monitored evidence supports a new decision.
- Audit a sample of auto-accepted documents.
- Version model, rule and threshold changes.
- Keep a manual fallback and an operational pause control.

