# GGUF Quantization: Pick the Right File for Your Machine

> Every GGUF quantization family explained, how to read a file name, work out what fits with context and slots, and quantize a model yourself with llama.cpp.

- Published: 2026-09-09
- Updated: 2026-11-15
- Author: Jacobo González Jaspe
- Language: en
- Tags: Quantization, GGUF, Ollama, Local AI, Hardware
- Canonical URL: https://vorluxai.com/en/blog/gguf-quantization-pick-the-right-file-for-your-hardware-en/
- Source: VORLUX AI (J4SGON S.L.), https://vorluxai.com/

---

By the end of this guide you can open a model's download page, pick the file that fits your machine and check the choice yourself. A laptop with 8 GB of memory is enough. Tool names and flags were checked against the llama.cpp and Ollama sources on 15 November 2026.

> **TL;DR**
> - You get: the four quantization families, a sizing rule backed by our measurements, and the commands to quantize a model yourself.
> - It costs: nothing to choose a file. Quantizing a 7B model took us 46.5 seconds on a CPU.
> - Verdict: start with Q4_K_M. Use Q8_0 if file plus context still fits. Go below 4 bits only to fit a much larger model.

## What you need

- A computer with 8 GB of memory or more and [Ollama](https://vorluxai.com/en/blog/install-ollama-first-local-llm-en/). Ours are 0.30.10 and 0.34.0.
- For step 6 only: a build of [llama.cpp](https://github.com/ggml-org/llama.cpp) ([1.0.0](https://github.com/ggml-org/llama.cpp/releases/tag/v1.0.0) came out on 13 November 2026).

## Step 1: read the file name

GGUF is the format that llama.cpp, Ollama and LM Studio load. Its [specification](https://github.com/ggml-org/ggml/blob/master/docs/gguf.md) fixes the naming, with the parts joined by hyphens:

```
[<Sidecar>]<BaseName><SizeLabel><FineTune><Version><Encoding><Type><Shard>.gguf
Grok-100B-v1.0-Q4_0-00003-of-00009.gguf
mmproj-Qwen2-VL-7B-v1.0-F16.gguf
```

- **SizeLabel** (`100B`): how many weights. Nothing else moves memory and speed as much.
- **Encoding** (`Q4_0`, `F16`): how each weight is stored. See step 2.
- **Shard** (`00003-of-00009`): the model is split and you need every file.
- **Sidecar** (`mmproj-`): an extra file for images or audio, not the model.

Uploaders add labels of their own. On [bartowski's Llama 3.3 70B page](https://huggingface.co/bartowski/Llama-3.3-70B-Instruct-GGUF), `Q4_K_L` is Q4_K_M with embedding and output weights at Q8_0; llama.cpp has no such type.

## Step 2: know the four families

| Family | Names | How weights are stored |
|---|---|---|
| Unquantized | `F32`, `F16`, `BF16` | Original floating point. The source you quantize from |
| Legacy | `Q4_0`, `Q5_0`, `Q8_0` | Blocks of 32 weights, rounded to nearest |
| K-quants | `Q2_K` to `Q6_K`, with `_S` `_M` `_L` | Blocks inside super-blocks, scales stored compactly |
| I-quants | `IQ1_S` to `IQ4_XS` | Super-blocks of 256 weights, tuned with an importance matrix |

The layouts come from [Hugging Face's GGUF reference](https://huggingface.co/docs/hub/gguf), which calls the legacy types "not used widely as of today". Yet `ollama show llava:7b` reports `Q4_0` on our machine.

**`_S`, `_M`, `_L`.** A K-quant file is a mix: a few sensitive tensors get more bits than the name says. Our Q4_K_M file from step 6 holds 169 tensors at Q4_K and 29 at Q6_K. `_S` promotes fewer.

**An importance matrix is not an I-quant.** It is calibration data: [`llama-imatrix`](https://github.com/ggml-org/llama.cpp/blob/master/tools/imatrix/README.md) runs sample text through the model and records which weights matter. K-quants can use one too ([PR 4930](https://github.com/ggml-org/llama.cpp/pull/4930)), through `--imatrix` in `llama-quantize`; bartowski's files all do. We did not build one; the commands are in step 6.

**I-quants are not free off the GPU.** bartowski's card says they run on CPU and Apple Metal but slower than the K-quant of similar size, and that they do not work with Vulkan. That advice sits on a card for an older model, so check your runtime. llama.cpp's own Llama 3.1 8B table puts IQ3_M at 70.15 tokens per second and Q3_K_M at 71.68, without naming the hardware. We have not timed an I-quant.

**Newer types.** The [tool](https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/quantize.cpp) also lists `TQ1_0` and `TQ2_0` for models trained with three-valued weights ([PR 8151](https://github.com/ggml-org/llama.cpp/pull/8151)), plus `MXFP4_MOE`, `Q1_0` and `Q2_0`. We have not used them.

## Step 3: know the sizes

| Type | Qwen2.5 7B, our dry run | Bits per weight, same run | Llama 3.1 8B, llama.cpp | Perplexity increase (Llama 3 8B) |
|---|---|---|---|---|
| IQ2_XS | 2,348.96 MiB | 2.59 | 2.42 GiB | not listed |
| Q2_K | 2,870.55 MiB | 3.16 | 2.95 GiB | +3.5199 |
| IQ3_M | 3,402.77 MiB | 3.75 | 3.52 GiB | not listed |
| Q3_K_M | 3,626.29 MiB | 3.99 | 3.74 GiB | +0.6569 |
| IQ4_XS | 4,047.72 MiB | 4.46 | 4.17 GiB | not listed |
| Q4_0 | 4,220.43 MiB | 4.65 | not listed | +0.4685 |
| Q4_K_S | 4,245.58 MiB | 4.68 | 4.36 GiB | +0.2689 |
| Q4_K_M | 4,460.45 MiB | 4.91 | 4.58 GiB | +0.1754 |
| Q5_K_S | 5,063.27 MiB | 5.58 | 5.21 GiB | +0.1049 |
| Q5_K_M | 5,186.92 MiB | 5.71 | 5.33 GiB | +0.0569 |
| Q6_K | 5,958.79 MiB | 6.56 | 6.14 GiB | +0.0217 |
| Q8_0 | 7,717.68 MiB | 8.50 | 7.95 GiB | +0.0026 |

Sources: our run, [gguf-quantize-run-2026-11-15.json](https://vorluxai.com/data/measurements/gguf-quantize-run-2026-11-15.json), where `llama-quantize --dry-run` printed each size without writing a file; llama.cpp's [quantize README](https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md) for the Llama 3.1 8B sizes; and the tool's type list for perplexity (both read 2026-11-15). The source file was 14,526.27 MiB at 16 bits. The I-quant rows were sized without an importance matrix, and IQ2_XS cannot be built without one.

Two things to read off it. Legacy `Q4_0` and `Q4_K_S` are 25 MiB apart, and the K-quant loses much less. Below Q4 the loss climbs fast.

llama.cpp prints MiB and GiB; Ollama and Hugging Face show decimal GB. For a model in no table: `size ≈ parameters × bits per weight ÷ 8`. Bits per weight is not a constant, because the mix of tensors changes with the model: llama.cpp's README gives Q4_K_M 4.8944 on Llama 3.1 8B, and our run printed 4.91 on Qwen2.5 7B. With 4.89, a 14B is about 8.6 GB.

## Step 4: work out whether it fits

A loaded model also reserves a cache for the conversation, once per parallel slot. Memory is the file, plus context times slots times a cost per token, plus 1 to 2 GB spare.

Measured 2 November 2026 with one 4.9 GB file, `llama3.1:8b` at Q4_K_M, on our two DGX Sparks (GB10, 128 GB unified memory, Ollama 0.30.10 and 0.34.0), as `ollama ps` reports it. Raw file: [ollama-num-ctx-2026-11-02.json](https://vorluxai.com/data/measurements/ollama-num-ctx-2026-11-02.json).

| Slots | `num_ctx` | Loaded size |
|---|---|---|
| 1 | 4,096 | 5.27 GB |
| 1 | 32,768 | 9.21 GB |
| 8 | 4,096 | 9.15 GB |
| 4 | 32,768 | 22.38 GB |
| 8 | 32,768 | 39.27 GB |

The same file took between 5.27 and 39.27 GB: slots times context is what counts. Not every model pays alike, as [Ollama num_ctx memory usage, measured](https://vorluxai.com/en/blog/ollama-context-length-memory-measured-en/) shows. The sum by hand is in [how much VRAM do I need?](https://vorluxai.com/en/blog/how-much-vram-local-llm-en/).

Set the context yourself. Ollama's [context length page](https://docs.ollama.com/context-length) (read 2026-11-15) gives 4k under 24 GiB of VRAM, 32k up to 48 GiB and 256k above, and its [FAQ](https://docs.ollama.com/faq) says parallel requests multiply it. On our second Spark that default, with eight slots, made loads fail until every model carried its own `num_ctx`.

**Shrinking the cache instead of the model.** Ollama's FAQ documents `OLLAMA_KV_CACHE_TYPE`. The default is `f16`; `q8_0` takes about half the cache memory and `q4_0` about a quarter. It needs Flash Attention, it applies to every model on the server, and the FAQ warns that some models, Qwen2 among them, may lose more precision. In llama.cpp the flags are `-ctk` and `-ctv`. It shrinks the context part only, never the file. We have not tested it.

Then run `ollama ps`: `PROCESSOR` must say `100% GPU`. If it shows a split such as the FAQ's `48%/52% CPU/GPU`, lower `num_ctx` first, then the quantization.

## Step 5: check which quantization Ollama serves

A plain tag hides the file. On our machine (Ollama 0.30.10, 2026-11-15), first block of the output:

```bash
ollama show llama3.1:8b
#   Model
#     architecture        llama
#     parameters          8.0B
#     context length      131072
#     embedding length    4096
#     quantization        Q4_K_M
```

In a script, read `details.quantization_level` from `/api/show`. Other files are on the [tags page](https://ollama.com/library/llama3.1/tags), from `8b-instruct-q2_K` to `8b-instruct-fp16`. Ollama also pulls GGUF files straight from Hugging Face. Per the [Hub guide](https://huggingface.co/docs/hub/ollama) (read 2026-11-15):

```bash
ollama run hf.co/<user>/<repo>          # Q4_K_M if the repository has it
ollama run hf.co/<user>/<repo>:Q8_0     # a named quantization, in either case
```

When a repository has no Q4_K_M, the Hub picks another file for you, so name the one you want; the full file name also works as the tag. We did not run these.

## Step 6: quantize a model yourself

Only when nobody publishes the file you want.

```bash
git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
cmake -B build && cmake --build build --config Release -j 8
python3 -m pip install -r requirements.txt

# phase 1: Hugging Face weights -> 16-bit GGUF
python3 convert_hf_to_gguf.py /path/to/model --outfile model-f16.gguf --outtype f16
# phase 2: size first, then quantize with 8 threads
./build/bin/llama-quantize --dry-run model-f16.gguf Q4_K_M
./build/bin/llama-quantize model-f16.gguf model-Q4_K_M.gguf Q4_K_M 8
```

We ran phase 2 only, with a July 2026 build. Our source was the FP16 file of Qwen2.5 7B that Ollama keeps on disk; `ollama show qwen2.5:7b-instruct-fp16 --modelfile` gives its path on the `FROM` line. On our first DGX Spark, CPU only:

```
llama_model_quantize_impl: model size  = 14526.27 MiB (16.00 BPW)
llama_model_quantize_impl: quant size  =  4460.45 MiB (4.91 BPW)
llama_quantize: quantize time = 46515.99 ms
```

A 15.24 GB file became 4.68 GB, the size Ollama lists for its own Q4_K_M tag. Read back, its file type is 15, `MOSTLY_Q4_K_M` in the specification. We did not load it. The process peaked at 14,885,452 kB of resident memory, about the size of the source file: the README says the model is read whole into memory, so budget RAM for the 16-bit file.

**With an importance matrix.** Needed for the smallest types (listed with the errors below) and optional for the rest. From the [imatrix README](https://github.com/ggml-org/llama.cpp/blob/master/tools/imatrix/README.md) on 2026-11-15:

```bash
# 1. run calibration text through the 16-bit model; writes imatrix.gguf
./build/bin/llama-imatrix -m model-f16.gguf -f calibration.txt -o imatrix.gguf -ngl 99
# 2. quantize with it
./build/bin/llama-quantize --imatrix imatrix.gguf model-f16.gguf model-IQ3_M.gguf IQ3_M 8
```

`-m` and `-f` are mandatory. `-ngl 99` offloads to the GPU, which the README recommends for speed; drop it on a CPU. `--chunks` caps how much text is processed. bartowski links the calibration text it uses on its model card. We ran neither command: our build has no `llama-imatrix`, and we kept this guide's runs on the CPU.

**Without building anything.** The quantize README points to the [GGUF-my-repo](https://huggingface.co/spaces/ggml-org/gguf-my-repo) Space on Hugging Face. It runs on their servers, so it is no route for weights that must stay in house. We have not used it.

To load it in Ollama: a Modelfile with `FROM ./model-Q4_K_M.gguf`, then `ollama create my-model`. We did not run that. Quantize first: the [import page](https://docs.ollama.com/import) (read 2026-11-15) says Ollama does not quantize GGUF files, and `-q` is for safetensors only in [current source](https://github.com/ollama/ollama/blob/main/cmd/cmd.go). To compare quality, see [llama-perplexity](https://github.com/ggml-org/llama.cpp/blob/master/tools/perplexity/README.md).

## What we measured

Qwen2.5 7B Instruct as three files, 5 November 2026, first DGX Spark, Ollama 0.30.10. Speed on a 278-word prompt; accuracy on 40 synthetic Spanish invoices. Raw files: [quantization-speed-2026-11-05.json](https://vorluxai.com/data/measurements/quantization-speed-2026-11-05.json) and [structured-extraction-2026-11-05.json](https://vorluxai.com/data/measurements/structured-extraction-2026-11-05.json).

| Encoding | File | Generation | Invoices fully right |
|---|---|---|---|
| Q4_K_M | 4.68 GB | 42.7 tok/s | 26 of 40 |
| Q8_0 | 8.10 GB | 28.2 tok/s | 34 of 40 |
| FP16 | 15.24 GB | 16.5 tok/s | 34 of 40 |

Q8_0 got eight more invoices right at 66% of Q4_K_M's speed. FP16 added nothing. One model, one task; the analysis is in [Q4_K_M vs Q8_0 vs FP16, measured](https://vorluxai.com/en/blog/q4-vs-q8-vs-fp16-measured-en/).

## What the spec sheet won't tell you

- **"requantizing from type q4_K is disabled".** We got it asking for Q3_K_M from a Q4_K_M file. Start from the 16-bit file: lost precision does not come back. `--allow-requantize` forces it. The dry run did not catch this.
- **"this quantization requires an importance matrix!".** We got it asking for `IQ2_XS` without one. The [source](https://github.com/ggml-org/llama.cpp/blob/master/src/llama-quant.cpp) requires it for the 1-bit and 2-bit I-quants, `IQ3_XXS` and `Q2_K_S`.
- **The dry run only warns.** For the same IQ2_XS request, `--dry-run` printed a size, exited 0 and said that completing it "will require an imatrix".
- **A split model opened at the wrong file.** bartowski splits anything over 50 GB, and its table marks the 49.95 GB Q5_K_M as split too. llama.cpp's [loader](https://github.com/ggml-org/llama.cpp/blob/master/src/llama-model-loader.cpp) raises `illegal split file idx` with the hint "model must be loaded with the first split". Keep every shard in one folder under its original name and point at `-00001-of-`. To get one file, `llama-gguf-split --merge first-shard.gguf merged.gguf` ([README](https://github.com/ggml-org/llama.cpp/blob/master/tools/gguf-split/README.md)). Ollama's import page takes a wildcard, `FROM /path/to/model-*.gguf`. The error text is from the source; we did not reproduce it.
- **llama.cpp's server counts differently.** There `-c` is a total that the slots share or split ([source](https://github.com/ggml-org/llama.cpp/blob/master/src/llama-context.cpp)). And `--fit` is on by default: per the [server README](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md) it adjusts the arguments you left unset to fit device memory, keeping 1,024 MiB free per device and never going under 4,096 tokens. So an unpinned context there is "as much as fits", where Ollama uses tiers. Pin it with `-c`. We did not run it.

## Decision table by memory

"Memory" is what the model can use: VRAM, or unified memory minus the system. "Loaded" sizes are ours, one slot unless stated; the rest are file sizes.

| Memory | Comfortable | Tight: short context, one slot | Skip, and why |
|---|---|---|---|
| 8 GB | 8B Q4_K_M (5.27 GB loaded, 4k) | 8B Q5_K_M (5.33 GiB) | 8B Q8_0 (7.95 GiB): no room left for context |
| 12 GB | 7B Q8_0, 4k (7.89 GB loaded) | 7B Q8_0, 32k (9.74 GB loaded) | 14B at 32k: ours loaded 15.29 GB |
| 16 GB | 14B Q4_K_M (8.99 GB) | 24B Q4_K_M (14.33 GB) | 7B FP16 (15.24 GB): same score as Q8_0 in our test |
| 24 GB | 26B Q4_K_M (17.47 GB loaded, 32k) | 32B Q4_K_M (about 19.6 GB by formula) | 70B IQ2_XXS (19.10 GB): bartowski rates it very low quality |
| 32 GB | 35B Q4_K_M (23.47 GB loaded, eight 32k slots) | 70B Q2_K (26.38 GB) | 70B Q3_K_S (30.91 GB): the card advises against it, and no room |
| 48 GB | 70B IQ4_XS (37.90 GB) | 70B Q4_K_M (42.52 GB) | 70B Q5_K_S (48.66 GB): the file alone fills it |
| 64 GB | 70B Q5_K_M (49.95 GB) | 70B Q6_K (57.89 GB) | 70B Q8_0 (74.98 GB): does not fit, and the card says it is rarely needed |

Context decides: our 14B loaded 15.29 GB at 32k. Our figures: [dgx-spark-review-prime-2026-11-02.json](https://vorluxai.com/data/measurements/dgx-spark-review-prime-2026-11-02.json), [mistral-small-24b-speed-ctx4k-2026-11-15.json](https://vorluxai.com/data/measurements/mistral-small-24b-speed-ctx4k-2026-11-15.json), [structured-extraction-2026-11-05.json](https://vorluxai.com/data/measurements/structured-extraction-2026-11-05.json) and the num_ctx file. The 8B and 70B sizes are llama.cpp's and bartowski's. We have not run a 70B: those rows leave one to two gigabytes spare, as bartowski advises.

## When not to do this

- **Do not quantize yourself** if a maintained repository already publishes the file. You save a build, a 16-bit download and the calibration step.
- **Do not go under 4 bits to save a little.** It is for fitting a model class you could not run otherwise, and then you test it.
- **Do not drop to Q4 for work where one wrong digit is a wrong answer** without measuring: in our invoice test it cost eight documents out of 40.
- **Do not requantize a quantized file.** Go back to the 16-bit source.
- **Do not change quantization to fix a memory problem before you have pinned `num_ctx` and the slots.** In step 4 those moved the same file from 5.27 to 39.27 GB.
- **Do not keep FP16 for serving.** Keep it as the source for new quantizations.

## Verdict

- **Buy: Q4_K_M** by default. The plain `llama3.1:8b` tag is that file.
- **Buy: Q8_0** for extraction where one character matters, if file plus context fits with room.
- **Wait: I-quants** until a larger model fits no other way. Then test it against the smaller model at Q4_K_M.
- **Skip: FP16** for running, and skip quantizing yourself when a maintained repository already has the file.

## FAQ

**Is Q4_K_M good enough?** It is the usual default: llama.cpp lists +0.1754 perplexity on Llama 3 8B. On our extraction task it still got 26 invoices right against 34 for Q8_0. It depends on the task, so run yours.

**Does a smaller file run faster?** In our test, yes: 42.7 tokens per second at Q4_K_M, 28.2 at Q8_0 and 16.5 at FP16. llama.cpp's Llama 3.1 8B table goes the same way (71.93, 50.93 and 29.17).

**A bigger model at fewer bits, or a smaller one at more?** We have not measured that. Perplexity figures for two different models cannot be compared, so the honest answer is to run your own task on both. The background is in [quantization explained: 70B models on consumer hardware](https://vorluxai.com/en/blog/quantization-explained-run-70b-models-consumer-hardware-en/).

**Can I turn a Q4 file back into Q8?** No. `llama-quantize` refuses, and forcing it gives a larger file with the Q4 errors still in it. Download the larger file.

**Do I need an importance matrix?** Only for the types that refuse to build without one. For K-quants it is optional, and bartowski's files already use one.

**Q4_K_M, Q4_K_S or Q4_0?** On our 7B, Q4_K_S saves 215 MiB over Q4_K_M and Q4_0 saves 240, for a perplexity increase of +0.2689 and +0.4685 instead of +0.1754 in llama.cpp's list. Take Q4_K_M unless those megabytes decide whether it fits.

**Why is the loaded size bigger than the file?** The context cache, once per slot. See step 4.

## Next steps

- Choose the model before the file: [best local LLMs by hardware and task](https://vorluxai.com/en/blog/best-local-llms-by-hardware-and-task-en/), or the [VRAM explorer](https://vorluxai.com/en/tools/vram-explorer/).
- Nothing fits? The [local AI hardware catalogue](https://vorluxai.com/en/blog/local-ai-hardware-catalog-2026-what-to-buy-at-each-budget-en/) goes by budget.
- Not sure local pays off? [Work out your break-even against the cloud](https://vorluxai.com/en/blog/cloud-vs-local-ai-cost-benchmarks-en/).
- Time two files on your machine: [benchmark a local model in 30 minutes](https://vorluxai.com/en/blog/benchmark-local-model-own-machine-30-minutes-en/).

## Work with us

We run this measurement on a client's own hardware before recommending a model. For a second opinion on a file or a machine, [get in touch](https://vorluxai.com/en/contact/) or read [how our consulting works](https://vorluxai.com/en/consulting/).
