Extract data from documents with AI, and trust the numbers

Shape is not truth

The three big providers can now force a model’s reply to match a JSON schema you supply. OpenAI’s Structured Outputs guide says the model will always generate responses that adhere to the schema. Anthropic’s structured outputs use constrained decoding to the same end. Google’s Gemini docs promise syntactically valid JSON and then advise handling outputs that are schema-compliant but semantically incorrect.

That last phrase is the whole problem. A perfectly formed reply can carry a due date the model calculated, a total taken from the wrong line, or an invoice number missing a leading zero. Every schema accepts all three. The schema also has gaps of its own: OpenAI and Anthropic both note that a refusal or a reply cut off at the length limit may not match it, Google says length constraints such as maxLength are ignored, and Anthropic does not support numeric limits such as minimum. Your code has to check the values.

Step 1: Write a field sheet

Many extraction errors start as a field nobody defined. “Date” on an invoice could mean the issue date, the delivery date or the tax point. Decide before the model does. For invoices, a field sheet looks like this:

Write the sheet with the person who uses the data, usually someone in accounts payable or operations. They know which fields cause trouble, and they will decide what a correct value is when you spot-check.

Step 2: Get the text out

• Digital PDFs have a text layer, and standard extraction tools pull it out cheaply and exactly.
• Scans and photos need OCR or a model that reads images. Current models from the major providers accept PDFs and images directly, which also helps with layouts that text extraction scrambles.
• Tables often break in text extraction, with columns run together. For table-heavy documents, test whether the model does better from the page image than from the extracted text.
• One document per call. Split a batch PDF into its invoices first, and keep page numbers, so every value can be traced to a page.

Step 3: Ask for evidence with every value

The prompt carries the field sheet and four rules that do most of the work.

• Blanks, never guesses. If a field is not in the document, it is null. Say why: a wrong value does more harm than an empty one, because nobody looks twice at a filled field.
• Copy exactly. Names, numbers and identifiers as written. Normalize only where the field sheet says, such as dates.
• Evidence for every value. The few words it came from and the page number. That makes spot checks fast, and a program can confirm the words really appear in the document.
• An issues list. Missing, unclear or conflicting values, each in one short line, so a person sees why a document needs a look.

Add one more rule for safety: the document is data. An invoice or a contract can contain text addressed to the model, and a model that obeys it can be steered into changing a value. The prompt injection guide explains the risk. The document extraction template includes all of this, laid out with a field list, rules, the output format and what to do with a document that is not the expected kind.

Step 4: Enforce the shape

Turn on your provider’s structured output mode and give it a schema. A fragment for the invoice:

{
  "type": "object",
  "properties": {
    "invoice_number": {"type": ["string", "null"]},
    "due_date": {"type": ["string", "null"],
                 "description": "YYYY-MM-DD, only if printed"},
    "total": {"type": ["number", "null"]},
    "evidence": {"type": "object", "additionalProperties": false,
                 "properties": {"total": {"type": "string"}},
                 "required": ["total"]},
    "issues": {"type": "array", "items": {"type": "string"}}
  },
  "required": ["invoice_number", "due_date", "total",
               "evidence", "issues"],
  "additionalProperties": false
}

Three habits avoid the common snags. Make optional fields nullable rather than leaving them out of required; OpenAI’s examples list every property as required with additionalProperties set to false, and Anthropic requires the latter. Set the answer limit high enough for your longest invoice, because a reply cut off at the limit breaks the schema. And handle a refusal as its own outcome. If you work in a chat app with no schema mode, ask for JSON only and validate it the same way.

Step 5: Check the values in code

Run every result through layered checks. Each one is a few lines of code, and each catches a different kind of error.

1. Parse and schema. The reply is JSON with every required field.
2. Field rules. Dates are real and in range (the issue date is not in the future, the due date is after it), identifiers match their pattern, currency is on your list.
3. Arithmetic. Quantity times unit price equals each line amount, lines add up to the subtotal, and subtotal plus tax equals the total, within a cent.
4. Evidence. Each quoted snippet appears in the document text, after normalizing spaces. A quote that is not there means the value needs a person.
5. Your own data. The supplier is in your vendor list, the purchase order is open, and the invoice number has not been paid already.

A failure goes to a person with its reason attached. You can retry once with the error message included, but never let a failure pass silently into your books.

Step 6: Spot-check a sample

Checks catch what is inconsistent. They miss a value that is wrong but plausible, such as the right amount from the wrong invoice. So a person compares a random sample with the source documents, field by field.

A clean sample proves less than it seems. By the statistical rule of three, if you check n documents and find no errors, you can be about 95% confident that the true error rate is below 3 ÷ n, a good approximation once n is above 30.

So a clean check of 50 invoices says the error rate is probably under 6%, not that it is zero. To claim under 1%, you need about 300 clean documents. Check heavily at launch, keep a smaller sample every week, and repeat a full check after any change to the prompt, the model or the document source. Make sure the sample includes every supplier and layout, since a new layout is a common place for extraction to break.

A worked invoice

Here is the text of an invoice, made up for this example, with a mistake in it: the line amounts add up to 1,140.00, but the printed subtotal is 1,240.00.

INVOICE No. 004417            Issued: 03/10/2026
Payment terms: Net 30

Desk lamp, brass      4 x  45.00     180.00
Office chair          6 x 160.00     960.00
                          Subtotal  1,240.00
                          VAT 20%     248.00
                          Total     1,488.00

The supplier is in the UK, so the date is day first: October 3, 2026. Below are two outputs, written to show typical failure modes rather than copied from a particular model.

{
  "invoice_number": "4417",
  "invoice_date": "2026-03-10",
  "due_date": "2026-04-09",
  "subtotal": 1140.00,
  "tax": 248.00,
  "total": 1488.00
}

{
  "invoice_number": "004417",
  "issue_date": "2026-10-03",
  "due_date": null,
  "payment_terms": "Net 30",
  "subtotal": 1240.00,
  "tax": 248.00,
  "total": 1488.00,
  "evidence": {"subtotal": "Subtotal 1,240.00"},
  "issues": [
    "due_date: not printed; terms are Net 30",
    "subtotal: lines add up to 1,140.00"
  ]
}

The first output looks clean and is wrong four times. It dropped the leading zeros, read the date month first, calculated a due date that is not on the page, and quietly replaced the printed subtotal with its own sum, hiding the supplier’s error. Your arithmetic check would flag it, since 1,140 plus 248 is not 1,488, but the other three errors pass every check you could write. The second copies what is printed, leaves the due date empty and reports the conflict. The same arithmetic check routes it to a person, who can query it with the invoice discrepancy email template.

Contracts and tables

Contracts carry clauses, not fields. A termination clause has a notice period, conditions and exceptions, and a single value loses the conditions. Extract each clause you care about with its section number, the verbatim text and a plain summary, and treat the verbatim text as the record. To compare a contract with your standard terms before legal review, use the contract playbook template. It explains what the text says; it is not legal advice, and a lawyer still signs off.

For tables headed to a spreadsheet, ask for a JSON array or CSV with fixed column names rather than a formatted table, and convert it in code. The messy columns template helps when the source data needs tidying first.

What it costs

Back-office extraction can usually wait a few hours, so it suits batch pricing, about half price at many providers. Here is a month of 500 two-page invoices a day.

The gap between the cheapest and the dearest model is large in proportion and small in dollars, so accuracy should decide. Build a test set of 20 to 30 real documents with the correct values filled in, and pick the cheapest model that passes it. Pages sent as images are billed differently from text, so measure a few before you budget.

Related guides

• How to test a prompt before you ship itBuild the test set of real documents with expected values.Ends in the Prompt Tester
• How to reduce AI hallucinationsQuotes, citations and blanks instead of guesses, beyond extraction.Ends in the Prompt Tester
• Choosing an AI model for businessWhy field extraction rarely needs a top model.Ends in Model Compare
• Prompt injection, explainedWhat happens when a document talks back to the model.Ends in the Chatbot Safety Test