View all articles
OllamaStructured outputBenchmarksInvoicesLocal AI

Ollama Structured Output with a JSON Schema: 40 Invoices Measured

JG
Jacobo González Jaspe
|
Abstract illustration: a minimal balance holding a navy cube and an amber sphere. Desde la spark
Illustration generated with AI on our own machine.
This article is also available in Spanish:Salida JSON estructurada con Ollama: 40 facturas, medido

By the end of this post you will get JSON out of a local model that your code can parse every time, and you will know how to check whether the values inside it are right. The 7B files we tested loaded in under 5 GB at Q4, so a 16 GB laptop is enough; our numbers come from a DGX Spark. We tested six models and files on 40 synthetic Spanish invoices to see where each one breaks.

The test

We wrote a generator, not a model, to produce 40 fictitious Spanish invoices from a fixed seed. Ten are formal invoices, ten are e-mails that announce one, ten are till receipts and ten are messy layouts with stray spaces and “N.I.F.”, the way a bad scan reads. Every document names the issuer and a client, each with its own tax ID.

Amounts come in Spanish formats (“1.234,56 €”, “1 234,56 EUR”) and dates in three styles. The model must return nine fields: invoice number, date, issuer name, issuer tax ID, client tax ID, base, VAT rate, VAT amount and total. The right answers come from the same generator, so no model wrote any of them. The companies are invented.

These are synthetic invoices, not real ones. That makes the right answers certain, but it also means we did not measure accuracy on real invoices. Expect your own documents to score differently, which is why Step 2 has you score ten of them.

We asked three ways, with the same Spanish instruction and temperature 0: a plain prompt, format: "json", and format set to a JSON schema.

What you need

  • Ollama 0.5 or newer. Schemas arrived in December 2024 (Ollama’s announcement). Our install guide covers a laptop.
  • A 7B or 8B model; qwen2.5:7b loaded in 4.8 GB with a 4k context in our run.
  • Ten real documents of your own and the right answers for them, written by a person. Without answers you only measure how confident the model sounds.
  • Python 3 with requests.

Step 1: write the schema

Name every field, give it a type and mark it required. Use number for money: a number field cannot hold a euro sign or a Spanish decimal comma.

JSON
{
  "type": "object",
  "properties": {
    "invoice_number": {"type": "string"},
    "issue_date":     {"type": "string", "description": "YYYY-MM-DD"},
    "issuer_name":    {"type": "string"},
    "issuer_tax_id":  {"type": "string"},
    "client_tax_id":  {"type": "string"},
    "base_amount":    {"type": "number"},
    "vat_rate":       {"type": "number"},
    "vat_amount":     {"type": "number"},
    "total_amount":   {"type": "number"}
  },
  "required": ["invoice_number", "issue_date", "issuer_name", "issuer_tax_id",
               "client_tax_id", "base_amount", "vat_rate", "vat_amount", "total_amount"]
}

Save it as schema.json. Keep the field list in your prompt as well: the schema fixes the shape, the prompt says what each field means.

Step 2: send it and score it

Python
import json, requests

SCHEMA = json.load(open("schema.json"))
PROMPT = ("Extract the invoice fields as JSON. issue_date as YYYY-MM-DD, amounts as numbers "
          "with a decimal point, tax IDs without spaces or dashes.\n\nDocument:\n")

def ok(field, got, gold):
    if field.endswith("_amount") or field == "vat_rate":
        return isinstance(got, (int, float)) and abs(got - gold) <= 0.01
    if field.endswith("tax_id"):
        return str(got).replace("-", "").replace(" ", "").upper() == gold
    return str(got).strip().casefold() == str(gold).strip().casefold()

docs = json.load(open("my-invoices.json"))   # [{"text": "...", "gold": {...}}, ...]
right = total = 0
for d in docs:
    r = requests.post("http://localhost:11434/api/chat", json={
        "model": "qwen2.5:7b", "stream": False, "format": SCHEMA,
        "options": {"temperature": 0, "num_ctx": 4096},
        "messages": [{"role": "user", "content": PROMPT + d["text"]}]},
        timeout=300).json()
    got = json.loads(r["message"]["content"])
    right += sum(ok(k, got.get(k), v) for k, v in d["gold"].items())
    total += len(d["gold"])
print(f"{right}/{total} fields correct")

Expected output: one line such as 329/360 fields correct, which is what Q4_K_M scored on our set with the bench’s Spanish prompt; this shorter prompt can score a few fields either way. If json.loads fails, run ollama --version: before 0.5 the schema in format is ignored. Our 40 invoices with their answers are public at structured-extraction-es-invoices-2026-11-05.json; load them with json.load(open(...))["docs"] to run your model on the same set first.

What we measured

Measured 2026-11-05 on our first DGX Spark (spark-43d5, NVIDIA GB10, 128 GB unified memory), Ollama 0.30.10, one model loaded at a time with num_ctx 4096 and seed 42. Raw file: structured-extraction-2026-11-05.json.

bash
python3 scripts/bench/run_for_topic.py --host prime --slug structured-extraction \
  --bench "structured_extraction.py --models llama3.1:8b,qwen2.5-coder:7b,qwen2.5:7b-instruct-q4_K_M,qwen2.5:7b-instruct-q8_0,qwen2.5:7b-instruct-fp16,gemma4:26b"

Schema mode, 40 invoices, 360 fields:

ModelFileLoadedFields rightInvoices fully rightE-mails fully rightSeconds per invoicetok/s
gemma4:26b (MoE)18.0 GB17.2 GB3604010 of 103.0762.3
qwen2.5:7b Q8_08.1 GB7.9 GB344345 of 104.9627.9
qwen2.5:7b FP1615.2 GB14.5 GB344345 of 108.3616.3
qwen2.5-coder:7b4.7 GB4.8 GB339334 of 103.4641.0
qwen2.5:7b Q4_K_M4.7 GB4.8 GB329262 of 103.3742.1
llama3.1:8b4.9 GB5.3 GB321221 of 102.8840.6

The three ways of asking (fields right of 360, and replies that were bare JSON your code can load directly):

ModelPlain promptformat: "json"format: schemaBare JSON: prompt / json / schema
gemma4:26b3603603600 / 40 / 40
qwen2.5:7b Q8_03413443445 / 40 / 40
qwen2.5:7b FP163413443443 / 40 / 40
qwen2.5-coder:7b3443393390 / 40 / 40
qwen2.5:7b Q4_K_M3323293296 / 40 / 40
llama3.1:8b3283213210 / 40 / 40

Without format, 226 of 240 replies came wrapped in a code fence or a sentence (“Here is the JSON…”). Every one parsed once we cut the fence out, but that is code you have to write and maintain. With format: "json", llama3.1:8b once returned all four numbers as strings; the schema run on the same invoice returned numbers. Changing the mode moved accuracy by 7 fields or fewer of 360.

What the spec sheet won’t tell you

  • Who issued it is the hard field. In the e-mails, the client’s details sit in the body and the issuer signs at the bottom. The 7B and 8B models got the issuer wrong in 5 to 9 of 10 e-mails, every time by naming the client instead. gemma4:26b never did.
  • Amounts were the easy part. All three Qwen2.5 7B Instruct files got base, VAT and total right on all 40 invoices in schema mode. The errors were names, tax IDs and one date.
  • Q4 drops letters from tax IDs. The Q4_K_M file returned a tax ID without its letter on six invoices where Q8_0 kept it; five of those were written in spaced groups, like “A 09 014 170” coming back as “09014170”. Q8_0 got 8 invoices fully right that Q4_K_M missed, and none the other way round.
  • “12/06/2026” became December. All three Qwen2.5 7B Instruct files read one till receipt’s date as month first. Ask for ISO dates and check the ones that land in the future.
  • The schema did not slow it down here. Schema replies were shorter than prompt replies (no fence, no preamble), so the seconds per invoice were lower for all six. llama3.1:8b was already loaded for another job on the box when our run started, so its timings may have been disturbed and are the least reliable in the table.

Verdict

  • Schema: buy (it is free). Send format with a schema on every extraction call. It is one field.
  • Model: measure first. On these synthetic invoices, gemma4:26b (17.2 GB loaded, so plan on 24 GB or more) was right on every field and took 3.07 s per invoice, faster than any Qwen2.5 7B file. On a 16 GB machine, Qwen2.5 7B at Q8_0 scored best of what we tested that fits, with a person reviewing issuer and tax IDs.
  • Skip extraction straight into your books from a 7B model with no review step. Even the best 7B file in our test got the issuer wrong on half the e-mailed invoices.

When not to do this

If your invoices arrive as scans or photos, you need text first: OCR or a vision model, which this test does not cover. We measured that separately with these same 40 invoices turned into images: read scanned invoices with a local vision model. If you receive structured e-invoices (Facturae XML, or the formats your invoicing software exports), parse the file; a language model adds errors to data that is already exact. And if you process a dozen invoices a month, typing them is faster than building and checking any of this.

Next steps

Work with us

We size the model and the machine by measuring, not by guessing. If you want to see your own documents going through a local model on real hardware, book a free 15-minute call or see how we work in consulting.

Diagram
Share: LinkedIn X
Veredicto semanal

Get new guides before anyone else

Subscribe and we tell you when new guides, templates and workflows go up. One email a week, no spam.

Already published: 102 guides and 25 templates. All free, no signup.

Bonus: the local-AI starter pack PDF when you subscribe
Once a week No spam Unsubscribe anytime

See what you get