# Q4_K_M vs Q8_0 vs FP16: The Quality Difference, Measured

> Qwen2.5 7B at Q4_K_M, Q8_0 and FP16 on 40 synthetic invoices: Q8_0 got 8 more fully right than Q4 at 66% of its speed. FP16 added nothing.

- Published: 2026-11-20
- Author: Jacobo González Jaspe
- Language: en
- Tags: Quantization, GGUF, Ollama, Benchmarks, Local AI
- Canonical URL: https://vorluxai.com/en/blog/q4-vs-q8-vs-fp16-measured-en/
- Source: VORLUX AI (J4SGON S.L.), https://vorluxai.com/

---

By the end of this post you will know which file to download for a 7B model, what each encoding costs in memory and speed, and what Q4_K_M actually gets wrong on a real task. The test takes an afternoon on any machine with about 10 GB free. We ran one model at three encodings on the same 40 invoices and the same speed prompt.

If you want to learn to read a GGUF file name and pick a file for your memory, that is our [GGUF guide](https://vorluxai.com/en/blog/gguf-quantization-pick-the-right-file-for-your-hardware-en/). This post is the measurement: one model, three files, one task with right answers.

> **TL;DR**
> - You get: Qwen2.5-7B-Instruct at Q4_K_M, Q8_0 and FP16, measured for speed, loaded memory and accuracy on 360 invoice fields.
> - It costs: Q8_0 loads 3.1 GB more than Q4_K_M and writes at 66% of its speed. FP16 needs another 6.6 GB and runs at 59% of Q8_0's speed.
> - Verdict: for extraction like this, Q8_0 if it fits your memory. It got 8 invoices fully right that Q4_K_M missed, and none the other way round. FP16 bought nothing.

## The question everyone answers with perplexity

Most pages on this topic quote perplexity, a score of how surprised a model is by a reference text. It is a fine research number. It does not tell you whether the tax ID on an invoice will come back with its letter.

So we used a task with right answers. Forty fictitious Spanish invoices, made by a generator from a fixed seed: formal invoices, e-mails, till receipts and messy scan-like layouts. The model returns nine fields as JSON (number, date, issuer, both tax IDs, base, VAT rate, VAT and total). The answers come from the generator, so no model wrote them. The invoices are synthetic: we did not measure accuracy on real ones, and our [structured output post](https://vorluxai.com/en/blog/ollama-structured-output-invoices-measured-en/) has the full six-model run on the same set. The set is public: [structured-extraction-es-invoices-2026-11-05.json](https://vorluxai.com/data/measurements/structured-extraction-es-invoices-2026-11-05.json).

## What you need

- Ollama, and enough free memory for the larger file you want to test (Q8_0 of a 7B model took 7.9 GB loaded at a 4k context, so leave about 10 GB).
- Two tags of the same model. In Ollama the encoding is in the tag name; the GGUF guide linked above explains how to read it.
- A task with right answers: ours, or ten of your own documents.

## Step 1: pull two encodings

```bash
ollama pull qwen2.5:7b-instruct-q4_K_M
ollama pull qwen2.5:7b-instruct-q8_0
ollama list | grep qwen2.5:7b-instruct
```

The plain tag `qwen2.5:7b` is the Q4_K_M file. Delete the one you do not keep afterwards (`ollama rm`); each is several gigabytes on disk.

## Step 2: measure speed and memory

```bash
for tag in q4_K_M q8_0; do
  curl -s localhost:11434/api/generate -d "{\"model\": \"qwen2.5:7b-instruct-$tag\", \"stream\": false,
    \"prompt\": \"Summarise the duties of a supplier contract in 150 words.\",
    \"options\": {\"num_predict\": 200, \"temperature\": 0, \"num_ctx\": 4096}}" \
  | python3 -c "import json,sys; d=json.load(sys.stdin); print('$tag', round(d['eval_count']/d['eval_duration']*1e9,1), 'tok/s')"
  ollama ps | grep qwen2.5
done
```

Run each twice and keep the second number: the first includes loading the file. `SIZE` in `ollama ps` is the memory the model holds, which depends on `num_ctx` too. It is rounded and counted in binary gigabytes: our FP16 run, 14.52 GB in the table below, shows as `14 GB`.

## Step 3: count what each one gets wrong

Run your task through both tags at temperature 0 and compare them **document by document**, not just by total score. Count two numbers: documents the small file got wrong and the big one got right, and the reverse. If the split is lopsided, the difference is real; if it is close to even, you are looking at noise. With 8 against 0, as we got, a chance split would happen less than once in a hundred tries (a two-sided sign test gives 0.008).

## What we measured

Measured 2026-11-05 on our first DGX Spark (`spark-43d5`, NVIDIA GB10, 128 GB unified memory), Ollama 0.30.10, one model resident at a time. Speed: our standard 278-word prompt, 200 tokens out, temperature 0, three runs, run 2 reported, default 32k context. Accuracy: the 40 invoices with a JSON schema in the request, temperature 0, `num_ctx` 4096. Raw files: [quantization-speed-2026-11-05.json](https://vorluxai.com/data/measurements/quantization-speed-2026-11-05.json) and [structured-extraction-2026-11-05.json](https://vorluxai.com/data/measurements/structured-extraction-2026-11-05.json).

```bash
python3 scripts/bench/run_for_topic.py --host prime --slug quantization-speed \
  --bench "local_model_benchmark.py --models qwen2.5:7b-instruct-q4_K_M,qwen2.5:7b-instruct-q8_0,qwen2.5:7b-instruct-fp16 --runs 3"
python3 scripts/bench/run_for_topic.py --host prime --slug structured-extraction \
  --bench "structured_extraction.py --models qwen2.5:7b-instruct-q4_K_M,qwen2.5:7b-instruct-q8_0,qwen2.5:7b-instruct-fp16"
```

The second command is shortened: the published run also covers three other models.

**Speed and memory:**

| Encoding | File | Loaded, 4k context | Loaded, 32k context | Generation | First token | Cold load |
|---|---|---|---|---|---|---|
| Q4_K_M | 4.68 GB | 4.75 GB | 6.59 GB | 42.7 tok/s | 0.28 s | 6.0 s |
| Q8_0 | 8.10 GB | 7.89 GB | 9.74 GB | 28.2 tok/s | 0.32 s | 5.5 s |
| FP16 | 15.24 GB | 14.52 GB | 16.36 GB | 16.5 tok/s | 0.33 s | 9.5 s |

**Accuracy on 40 invoices (schema mode):**

| Encoding | Fields right (of 360) | Invoices fully right | Tax IDs right (of 80) | Amounts right (of 120) | Seconds per invoice |
|---|---|---|---|---|---|
| Q4_K_M | 329 | 26 | 58 | 120 | 3.37 |
| Q8_0 | 344 | 34 | 70 | 120 | 4.96 |
| FP16 | 344 | 34 | 70 | 120 | 8.36 |

Two facts behind the tables. Q8_0 and FP16 gave byte-identical replies on 118 of 120 calls across our three ways of asking. Q4_K_M returned the same data as Q8_0 on 31 of 40 invoices in schema mode (27 byte for byte). Of the nine where the data differed, Q8_0 got more fields right on eight; the ninth was a tie, with both wrong.

## What the spec sheet won't tell you

- **Q4 loses characters, not arithmetic.** All three files got base, VAT and total right on every invoice. Q4_K_M's 15 extra wrong fields were all names and tax IDs. It dropped the letter from a tax ID on six invoices, mostly ones written in spaced groups ("A 09 014 170" came back as "09014170"), and on three more e-mails it took the client for the issuer. A tax ID with a missing letter fails every check downstream.
- **The bigger file did not fix reading.** Q8_0 and FP16 still named the client as the issuer on 5 of 10 e-mailed invoices, the same five. More bits did not make the model understand who signed the e-mail.
- **Speed follows the file size.** On this machine generation speed fell as the file grew, a little less than in proportion: 42.7, 28.2 and 16.5 tok/s for 4.7, 8.1 and 15.2 GB. Each step up was 1.7 to 1.9 times the bytes and 1.5 to 1.7 times slower. Time to first token barely moved.
- **On a GPU with less memory than the file, none of this applies.** If the model spills to the CPU, expect the larger file to be far slower than these numbers (we did not measure that case). Check that `ollama ps` says `100% GPU`.

## Verdict

- **Buy Q8_0** for field extraction like ours, where one character matters, when the file plus your context fits with room to spare. We did not test code or chat.
- **Keep Q4_K_M** when memory is tight (8 GB or less) or speed matters more than exact strings: it is 1.5 times faster, and on this task its extra errors were all in names and tax IDs.
- **Skip FP16** for this 7B model. Nearly twice the file size of Q8_0, 59% of its speed, and the same answers on 118 of 120 calls.

## When not to do this

If your machine fits only Q4_K_M, the comparison is academic: a smaller model at Q8_0 is the other option, and that is a different test. If you use the model for conversation, the difference we measured may not show at all; ten of your own prompts side by side will tell you faster. And this is one model on one task. A larger model or another family can lose more, or less, at 4 bits; measure yours before you standardise on a file.

## Next steps

- Read a GGUF file name and estimate sizes before you pull: [GGUF quantization: pick the right file](https://vorluxai.com/en/blog/gguf-quantization-pick-the-right-file-for-your-hardware-en/).
- Get JSON your code can always parse, and score it: [Ollama structured output, 40 invoices measured](https://vorluxai.com/en/blog/ollama-structured-output-invoices-measured-en/).
- Add the context to the file size before you buy: [how much VRAM do I need?](https://vorluxai.com/en/blog/how-much-vram-local-llm-en/).
- See where the memory goes as context grows: [Ollama num_ctx memory, measured](https://vorluxai.com/en/blog/ollama-context-length-memory-measured-en/).

## Work with us

We size the model and the machine by measuring, not by guessing. If you want to see your own task running on real hardware, [book a free 15-minute call](https://vorluxai.com/en/contact/) or [see how we work in consulting](https://vorluxai.com/en/consulting/).
