By the end of this guide you can open a model’s download page, pick the file that fits your machine and check the choice yourself. A laptop with 8 GB of memory is enough. Tool names and flags were checked against the llama.cpp and Ollama sources on 15 November 2026.
What you need
- A computer with 8 GB of memory or more and Ollama. Ours are 0.30.10 and 0.34.0.
- For step 6 only: a build of llama.cpp (1.0.0 came out on 13 November 2026).
Step 1: read the file name
GGUF is the format that llama.cpp, Ollama and LM Studio load. Its specification fixes the naming, with the parts joined by hyphens:
[<Sidecar>]<BaseName><SizeLabel><FineTune><Version><Encoding><Type><Shard>.gguf
Grok-100B-v1.0-Q4_0-00003-of-00009.gguf
mmproj-Qwen2-VL-7B-v1.0-F16.gguf- SizeLabel (
100B): how many weights. Nothing else moves memory and speed as much. - Encoding (
Q4_0,F16): how each weight is stored. See step 2. - Shard (
00003-of-00009): the model is split and you need every file. - Sidecar (
mmproj-): an extra file for images or audio, not the model.
Uploaders add labels of their own. On bartowski’s Llama 3.3 70B page, Q4_K_L is Q4_K_M with embedding and output weights at Q8_0; llama.cpp has no such type.
Step 2: know the four families
| Family | Names | How weights are stored |
|---|---|---|
| Unquantized | F32, F16, BF16 | Original floating point. The source you quantize from |
| Legacy | Q4_0, Q5_0, Q8_0 | Blocks of 32 weights, rounded to nearest |
| K-quants | Q2_K to Q6_K, with _S _M _L | Blocks inside super-blocks, scales stored compactly |
| I-quants | IQ1_S to IQ4_XS | Super-blocks of 256 weights, tuned with an importance matrix |
The layouts come from Hugging Face’s GGUF reference, which calls the legacy types “not used widely as of today”. Yet ollama show llava:7b reports Q4_0 on our machine.
_S, _M, _L. A K-quant file is a mix: a few sensitive tensors get more bits than the name says. Our Q4_K_M file from step 6 holds 169 tensors at Q4_K and 29 at Q6_K. _S promotes fewer.
An importance matrix is not an I-quant. It is calibration data: llama-imatrix runs sample text through the model and records which weights matter. K-quants can use one too (PR 4930), through --imatrix in llama-quantize; bartowski’s files all do. We did not build one; the commands are in step 6.
I-quants are not free off the GPU. bartowski’s card says they run on CPU and Apple Metal but slower than the K-quant of similar size, and that they do not work with Vulkan. That advice sits on a card for an older model, so check your runtime. llama.cpp’s own Llama 3.1 8B table puts IQ3_M at 70.15 tokens per second and Q3_K_M at 71.68, without naming the hardware. We have not timed an I-quant.
Newer types. The tool also lists TQ1_0 and TQ2_0 for models trained with three-valued weights (PR 8151), plus MXFP4_MOE, Q1_0 and Q2_0. We have not used them.
Step 3: know the sizes
| Type | Qwen2.5 7B, our dry run | Bits per weight, same run | Llama 3.1 8B, llama.cpp | Perplexity increase (Llama 3 8B) |
|---|---|---|---|---|
| IQ2_XS | 2,348.96 MiB | 2.59 | 2.42 GiB | not listed |
| Q2_K | 2,870.55 MiB | 3.16 | 2.95 GiB | +3.5199 |
| IQ3_M | 3,402.77 MiB | 3.75 | 3.52 GiB | not listed |
| Q3_K_M | 3,626.29 MiB | 3.99 | 3.74 GiB | +0.6569 |
| IQ4_XS | 4,047.72 MiB | 4.46 | 4.17 GiB | not listed |
| Q4_0 | 4,220.43 MiB | 4.65 | not listed | +0.4685 |
| Q4_K_S | 4,245.58 MiB | 4.68 | 4.36 GiB | +0.2689 |
| Q4_K_M | 4,460.45 MiB | 4.91 | 4.58 GiB | +0.1754 |
| Q5_K_S | 5,063.27 MiB | 5.58 | 5.21 GiB | +0.1049 |
| Q5_K_M | 5,186.92 MiB | 5.71 | 5.33 GiB | +0.0569 |
| Q6_K | 5,958.79 MiB | 6.56 | 6.14 GiB | +0.0217 |
| Q8_0 | 7,717.68 MiB | 8.50 | 7.95 GiB | +0.0026 |
Sources: our run, gguf-quantize-run-2026-11-15.json, where llama-quantize --dry-run printed each size without writing a file; llama.cpp’s quantize README for the Llama 3.1 8B sizes; and the tool’s type list for perplexity (both read 2026-11-15). The source file was 14,526.27 MiB at 16 bits. The I-quant rows were sized without an importance matrix, and IQ2_XS cannot be built without one.
Two things to read off it. Legacy Q4_0 and Q4_K_S are 25 MiB apart, and the K-quant loses much less. Below Q4 the loss climbs fast.
llama.cpp prints MiB and GiB; Ollama and Hugging Face show decimal GB. For a model in no table: size ≈ parameters × bits per weight ÷ 8. Bits per weight is not a constant, because the mix of tensors changes with the model: llama.cpp’s README gives Q4_K_M 4.8944 on Llama 3.1 8B, and our run printed 4.91 on Qwen2.5 7B. With 4.89, a 14B is about 8.6 GB.
Step 4: work out whether it fits
A loaded model also reserves a cache for the conversation, once per parallel slot. Memory is the file, plus context times slots times a cost per token, plus 1 to 2 GB spare.
Measured 2 November 2026 with one 4.9 GB file, llama3.1:8b at Q4_K_M, on our two DGX Sparks (GB10, 128 GB unified memory, Ollama 0.30.10 and 0.34.0), as ollama ps reports it. Raw file: ollama-num-ctx-2026-11-02.json.
| Slots | num_ctx | Loaded size |
|---|---|---|
| 1 | 4,096 | 5.27 GB |
| 1 | 32,768 | 9.21 GB |
| 8 | 4,096 | 9.15 GB |
| 4 | 32,768 | 22.38 GB |
| 8 | 32,768 | 39.27 GB |
The same file took between 5.27 and 39.27 GB: slots times context is what counts. Not every model pays alike, as Ollama num_ctx memory usage, measured shows. The sum by hand is in how much VRAM do I need?.
Set the context yourself. Ollama’s context length page (read 2026-11-15) gives 4k under 24 GiB of VRAM, 32k up to 48 GiB and 256k above, and its FAQ says parallel requests multiply it. On our second Spark that default, with eight slots, made loads fail until every model carried its own num_ctx.
Shrinking the cache instead of the model. Ollama’s FAQ documents OLLAMA_KV_CACHE_TYPE. The default is f16; q8_0 takes about half the cache memory and q4_0 about a quarter. It needs Flash Attention, it applies to every model on the server, and the FAQ warns that some models, Qwen2 among them, may lose more precision. In llama.cpp the flags are -ctk and -ctv. It shrinks the context part only, never the file. We have not tested it.
Then run ollama ps: PROCESSOR must say 100% GPU. If it shows a split such as the FAQ’s 48%/52% CPU/GPU, lower num_ctx first, then the quantization.
Step 5: check which quantization Ollama serves
A plain tag hides the file. On our machine (Ollama 0.30.10, 2026-11-15), first block of the output:
ollama show llama3.1:8b
# Model
# architecture llama
# parameters 8.0B
# context length 131072
# embedding length 4096
# quantization Q4_K_MIn a script, read details.quantization_level from /api/show. Other files are on the tags page, from 8b-instruct-q2_K to 8b-instruct-fp16. Ollama also pulls GGUF files straight from Hugging Face. Per the Hub guide (read 2026-11-15):
ollama run hf.co/<user>/<repo> # Q4_K_M if the repository has it
ollama run hf.co/<user>/<repo>:Q8_0 # a named quantization, in either caseWhen a repository has no Q4_K_M, the Hub picks another file for you, so name the one you want; the full file name also works as the tag. We did not run these.
Step 6: quantize a model yourself
Only when nobody publishes the file you want.
git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
cmake -B build && cmake --build build --config Release -j 8
python3 -m pip install -r requirements.txt
# phase 1: Hugging Face weights -> 16-bit GGUF
python3 convert_hf_to_gguf.py /path/to/model --outfile model-f16.gguf --outtype f16
# phase 2: size first, then quantize with 8 threads
./build/bin/llama-quantize --dry-run model-f16.gguf Q4_K_M
./build/bin/llama-quantize model-f16.gguf model-Q4_K_M.gguf Q4_K_M 8We ran phase 2 only, with a July 2026 build. Our source was the FP16 file of Qwen2.5 7B that Ollama keeps on disk; ollama show qwen2.5:7b-instruct-fp16 --modelfile gives its path on the FROM line. On our first DGX Spark, CPU only:
llama_model_quantize_impl: model size = 14526.27 MiB (16.00 BPW)
llama_model_quantize_impl: quant size = 4460.45 MiB (4.91 BPW)
llama_quantize: quantize time = 46515.99 msA 15.24 GB file became 4.68 GB, the size Ollama lists for its own Q4_K_M tag. Read back, its file type is 15, MOSTLY_Q4_K_M in the specification. We did not load it. The process peaked at 14,885,452 kB of resident memory, about the size of the source file: the README says the model is read whole into memory, so budget RAM for the 16-bit file.
With an importance matrix. Needed for the smallest types (listed with the errors below) and optional for the rest. From the imatrix README on 2026-11-15:
# 1. run calibration text through the 16-bit model; writes imatrix.gguf
./build/bin/llama-imatrix -m model-f16.gguf -f calibration.txt -o imatrix.gguf -ngl 99
# 2. quantize with it
./build/bin/llama-quantize --imatrix imatrix.gguf model-f16.gguf model-IQ3_M.gguf IQ3_M 8-m and -f are mandatory. -ngl 99 offloads to the GPU, which the README recommends for speed; drop it on a CPU. --chunks caps how much text is processed. bartowski links the calibration text it uses on its model card. We ran neither command: our build has no llama-imatrix, and we kept this guide’s runs on the CPU.
Without building anything. The quantize README points to the GGUF-my-repo Space on Hugging Face. It runs on their servers, so it is no route for weights that must stay in house. We have not used it.
To load it in Ollama: a Modelfile with FROM ./model-Q4_K_M.gguf, then ollama create my-model. We did not run that. Quantize first: the import page (read 2026-11-15) says Ollama does not quantize GGUF files, and -q is for safetensors only in current source. To compare quality, see llama-perplexity.
What we measured
Qwen2.5 7B Instruct as three files, 5 November 2026, first DGX Spark, Ollama 0.30.10. Speed on a 278-word prompt; accuracy on 40 synthetic Spanish invoices. Raw files: quantization-speed-2026-11-05.json and structured-extraction-2026-11-05.json.
| Encoding | File | Generation | Invoices fully right |
|---|---|---|---|
| Q4_K_M | 4.68 GB | 42.7 tok/s | 26 of 40 |
| Q8_0 | 8.10 GB | 28.2 tok/s | 34 of 40 |
| FP16 | 15.24 GB | 16.5 tok/s | 34 of 40 |
Q8_0 got eight more invoices right at 66% of Q4_K_M’s speed. FP16 added nothing. One model, one task; the analysis is in Q4_K_M vs Q8_0 vs FP16, measured.
What the spec sheet won’t tell you
- “requantizing from type q4_K is disabled”. We got it asking for Q3_K_M from a Q4_K_M file. Start from the 16-bit file: lost precision does not come back.
--allow-requantizeforces it. The dry run did not catch this. - “this quantization requires an importance matrix!”. We got it asking for
IQ2_XSwithout one. The source requires it for the 1-bit and 2-bit I-quants,IQ3_XXSandQ2_K_S. - The dry run only warns. For the same IQ2_XS request,
--dry-runprinted a size, exited 0 and said that completing it “will require an imatrix”. - A split model opened at the wrong file. bartowski splits anything over 50 GB, and its table marks the 49.95 GB Q5_K_M as split too. llama.cpp’s loader raises
illegal split file idxwith the hint “model must be loaded with the first split”. Keep every shard in one folder under its original name and point at-00001-of-. To get one file,llama-gguf-split --merge first-shard.gguf merged.gguf(README). Ollama’s import page takes a wildcard,FROM /path/to/model-*.gguf. The error text is from the source; we did not reproduce it. - llama.cpp’s server counts differently. There
-cis a total that the slots share or split (source). And--fitis on by default: per the server README it adjusts the arguments you left unset to fit device memory, keeping 1,024 MiB free per device and never going under 4,096 tokens. So an unpinned context there is “as much as fits”, where Ollama uses tiers. Pin it with-c. We did not run it.
Decision table by memory
“Memory” is what the model can use: VRAM, or unified memory minus the system. “Loaded” sizes are ours, one slot unless stated; the rest are file sizes.
| Memory | Comfortable | Tight: short context, one slot | Skip, and why |
|---|---|---|---|
| 8 GB | 8B Q4_K_M (5.27 GB loaded, 4k) | 8B Q5_K_M (5.33 GiB) | 8B Q8_0 (7.95 GiB): no room left for context |
| 12 GB | 7B Q8_0, 4k (7.89 GB loaded) | 7B Q8_0, 32k (9.74 GB loaded) | 14B at 32k: ours loaded 15.29 GB |
| 16 GB | 14B Q4_K_M (8.99 GB) | 24B Q4_K_M (14.33 GB) | 7B FP16 (15.24 GB): same score as Q8_0 in our test |
| 24 GB | 26B Q4_K_M (17.47 GB loaded, 32k) | 32B Q4_K_M (about 19.6 GB by formula) | 70B IQ2_XXS (19.10 GB): bartowski rates it very low quality |
| 32 GB | 35B Q4_K_M (23.47 GB loaded, eight 32k slots) | 70B Q2_K (26.38 GB) | 70B Q3_K_S (30.91 GB): the card advises against it, and no room |
| 48 GB | 70B IQ4_XS (37.90 GB) | 70B Q4_K_M (42.52 GB) | 70B Q5_K_S (48.66 GB): the file alone fills it |
| 64 GB | 70B Q5_K_M (49.95 GB) | 70B Q6_K (57.89 GB) | 70B Q8_0 (74.98 GB): does not fit, and the card says it is rarely needed |
Context decides: our 14B loaded 15.29 GB at 32k. Our figures: dgx-spark-review-prime-2026-11-02.json, mistral-small-24b-speed-ctx4k-2026-11-15.json, structured-extraction-2026-11-05.json and the num_ctx file. The 8B and 70B sizes are llama.cpp’s and bartowski’s. We have not run a 70B: those rows leave one to two gigabytes spare, as bartowski advises.
When not to do this
- Do not quantize yourself if a maintained repository already publishes the file. You save a build, a 16-bit download and the calibration step.
- Do not go under 4 bits to save a little. It is for fitting a model class you could not run otherwise, and then you test it.
- Do not drop to Q4 for work where one wrong digit is a wrong answer without measuring: in our invoice test it cost eight documents out of 40.
- Do not requantize a quantized file. Go back to the 16-bit source.
- Do not change quantization to fix a memory problem before you have pinned
num_ctxand the slots. In step 4 those moved the same file from 5.27 to 39.27 GB. - Do not keep FP16 for serving. Keep it as the source for new quantizations.
Verdict
- Buy: Q4_K_M by default. The plain
llama3.1:8btag is that file. - Buy: Q8_0 for extraction where one character matters, if file plus context fits with room.
- Wait: I-quants until a larger model fits no other way. Then test it against the smaller model at Q4_K_M.
- Skip: FP16 for running, and skip quantizing yourself when a maintained repository already has the file.
FAQ
Is Q4_K_M good enough? It is the usual default: llama.cpp lists +0.1754 perplexity on Llama 3 8B. On our extraction task it still got 26 invoices right against 34 for Q8_0. It depends on the task, so run yours.
Does a smaller file run faster? In our test, yes: 42.7 tokens per second at Q4_K_M, 28.2 at Q8_0 and 16.5 at FP16. llama.cpp’s Llama 3.1 8B table goes the same way (71.93, 50.93 and 29.17).
A bigger model at fewer bits, or a smaller one at more? We have not measured that. Perplexity figures for two different models cannot be compared, so the honest answer is to run your own task on both. The background is in quantization explained: 70B models on consumer hardware.
Can I turn a Q4 file back into Q8? No. llama-quantize refuses, and forcing it gives a larger file with the Q4 errors still in it. Download the larger file.
Do I need an importance matrix? Only for the types that refuse to build without one. For K-quants it is optional, and bartowski’s files already use one.
Q4_K_M, Q4_K_S or Q4_0? On our 7B, Q4_K_S saves 215 MiB over Q4_K_M and Q4_0 saves 240, for a perplexity increase of +0.2689 and +0.4685 instead of +0.1754 in llama.cpp’s list. Take Q4_K_M unless those megabytes decide whether it fits.
Why is the loaded size bigger than the file? The context cache, once per slot. See step 4.
Next steps
- Choose the model before the file: best local LLMs by hardware and task, or the VRAM explorer.
- Nothing fits? The local AI hardware catalogue goes by budget.
- Not sure local pays off? Work out your break-even against the cloud.
- Time two files on your machine: benchmark a local model in 30 minutes.
Work with us
We run this measurement on a client’s own hardware before recommending a model. For a second opinion on a file or a machine, get in touch or read how our consulting works.