GGUF Quantization: Pick the Right Model File for Your 8, 16, 24 or 32 GB Machine

By the end of this post you will be able to look at a model download page, pick the one file that fits your machine, and prove with your own numbers that it was the right pick. It works on any laptop with 8 GB of RAM; the commands are the same on a EUR 250 board and on our workstation.

What you need

• A computer with 8 GB of memory or more. A GPU helps but is not required.
• Ollama installed (free, no account). Everything below also applies to llama.cpp and LM Studio, which read the same files.
• Fifteen minutes and about 5 GB of disk for one model download.
• Optional: an AI assistant open next to you. At the end there is a prompt you can paste into it with your own numbers.

Step 1: Read the file name

GGUF is the file format llama.cpp and Ollama use. The ggml specification describes it as “a binary format that is designed for fast loading and saving of models”, and defines the naming scheme:

<BaseName><SizeLabel><FineTune><Version><Encoding><Type><Shard>.gguf
Hermes-2-Pro-Llama-3-8B-F16.gguf
Grok-100B-v1.0-Q4_0-00003-of-00009.gguf

Three parts matter to you:

• SizeLabel (8B, 70B): the parameter count. Bigger is slower and needs more memory, whatever the encoding.
• Encoding (Q4_K_M, Q8_0, F16): how many bits each weight is stored in. F16 is the original 16-bit weights. Q8_0 is 8-bit, Q4_K_M is 4-bit with the “K” block method and “M” (medium) mix. Per the Hugging Face GGUF documentation, Q4_K costs 4.5 bits per weight and Q5_K costs 5.5; the _S, _M, _L suffixes keep a few sensitive layers at higher precision. IQ types (IQ4_XS, IQ2_M) use an importance matrix to squeeze a bit further.
• Shard (00003-of-00009): the model is split across files. You need all of them.

Ollama hides the file behind a tag. ollama pull llama3.1:8b gives you a Q4_K_M file; the page ollama.com/library/llama3.1/tags lists the alternatives, from 8b-instruct-q2_K to 8b-instruct-q8_0 and 8b-instruct-fp16. You can check what you have:

ollama show llama3.1:8b
# parameters 8.0B · context length 131072 · quantization Q4_K_M

Step 2: Know the sizes

The llama.cpp project publishes the file size of every quantization of Llama 3.1 8B in its quantize README; the 70B column comes from the bartowski Llama 3.3 70B GGUF table.

Encoding	Bits per weight	Llama 3.1 8B	Llama 3.3 70B	What the community says about quality
Q2_K	3.16	2.95 GiB	26.4 GB	very low, “surprisingly usable”
Q3_K_M	4.00	3.74 GiB	34.3 GB	low
IQ4_XS	~4.25	—	37.9 GB	decent, close to Q4_K_S
Q4_K_M	4.89	4.58 GiB	42.5 GB	good, the default
Q5_K_M	5.70	5.33 GiB	50.0 GB	high
Q6_K	6.56	6.14 GiB	57.9 GB	very high, near the original
Q8_0	8.50	7.95 GiB	75.0 GB	practically the original
F16	16.0	14.96 GiB	~141 GB	the original

For a model that is not in the table, estimate the file as parameters × bits per weight ÷ 8. A 14B model at Q4_K_M is about 14 × 4.89 ÷ 8 = 8.6 GB; a 32B model at Q4_K_M is about 19.6 GB.

Step 3: Add the context window, then match your memory

The file is not the whole story. The model also keeps a cache of everything in the current conversation, and its size depends on the context window you allow, not on the quantization. We measured this on our workstation (NVIDIA GB10, 128 GB unified memory, Ollama 0.30.10, 2026-09-09) with the same 4.9 GB llama3.1:8b file:

num_ctx	Resident memory (ollama ps)	Generation speed
4,096 tokens	5.3 GB	23.8, 23.2 and 15.8 tokens/s on three runs
131,072 tokens	22 GB	20.0 and 24.0 tokens/s on two runs

Same file, four times the memory. The rule that falls out of this: budget the file size plus 1 GB for a 4k context, then about 1 GB more for every extra 8k tokens on an 8B model, and more on bigger models. The 15.8 tokens/s run happened while another service loaded a second model on the same shared machine; the day before, with the box quiet, the same model gave 37.8 tokens/s. Your numbers will vary with load, which is exactly why you measure rather than trust a table.

With that rule, here is what fits where, leaving headroom for the operating system:

Your memory (GPU VRAM or unified)	Comfortable pick	Possible with care	Do not bother
8 GB	8B at Q4_K_M, 3B at Q8_0	8B at Q5_K_M with a 4k context	8B at Q8_0, anything 14B+
16 GB	8B at Q8_0, 14B at Q4_K_M	14B at Q5_K_M	32B
24 GB	14B at Q8_0, 32B at Q4_K_M	32B at Q5_K_M with a short context	70B at any encoding (IQ2_M is 24.1 GB before context)
32 GB	32B at Q5_K_M	32B at Q6_K	70B below IQ4_XS is not worth the quality loss
48 to 64 GB	70B at IQ4_XS or Q4_K_M	70B at Q5_K_M on 64 GB	70B at Q8_0

Step 4: Measure speed and memory yourself

Pull two encodings of the same model and time them with the same prompt. Ollama returns the token count and the generation time in nanoseconds:

ollama pull llama3.1:8b-instruct-q4_K_M
ollama pull llama3.1:8b-instruct-q8_0

for tag in 8b-instruct-q4_K_M 8b-instruct-q8_0; do
  curl -s localhost:11434/api/generate -d "{
    \"model\": \"llama3.1:$tag\", \"stream\": false,
    \"prompt\": \"Summarise the main obligations of a supplier contract in 200 words.\",
    \"options\": {\"num_predict\": 300, \"num_ctx\": 4096, \"temperature\": 0.2}
  }" | python3 -c "import json,sys; d=json.load(sys.stdin); \
    print('$tag', round(d['eval_count']/d['eval_duration']*1e9,1), 'tok/s')"
  ollama ps   # SIZE column = resident memory for this tag
done

Expected output: two lines with tokens per second and two ollama ps tables. If the PROCESSOR column says anything other than 100% GPU on a machine with a GPU, the file did not fit and part of it is running on the CPU; drop one encoding level or reduce num_ctx.

Step 5: Measure quality, not just speed

Speed and memory are easy to read; quality needs a small test of your own. Two methods, from cheap to rigorous:

1. Ten of your own prompts. Save the ten questions your team really asks (a contract clause, a product description, a SQL query). Run them through both tags with temperature: 0 and put the answers side by side. If you cannot tell Q4_K_M from Q8_0 on your tasks, keep the smaller file and the memory it frees.
2. Perplexity with llama.cpp. If you build llama.cpp, llama-perplexity -m model.gguf -f your-text.txt gives a single number per file; lower is better, and the gap between encodings on your own documents is the number that matters.

Prompt for your assistant, with your numbers pasted in: “Here are tokens per second, resident memory and ten paired answers for Q4_K_M and Q8_0. Which encoding should I keep for a 16 GB machine that serves five people, and why?”

Where this fits

• A 4-bit file is the right default. It is what Ollama gives you when you type a model name, and it is what most published benchmarks are run on.
• Go up to Q6_K or Q8_0 when the file still fits with your working context. Code and structured extraction are where the extra bits are most often noticed.
• Go down to Q3 or IQ types only to fit a much larger model. A 70B model at IQ4_XS on a 48 GB machine is a good trade; an 8B model at Q2_K on an 8 GB machine is not, because a 3B model at Q8_0 will usually answer better.

Honest limit: quantization does not change what the model knows or how long its context is. It changes only how faithfully the weights are stored and how much memory they take.

Next steps

• The wider picture on why 4-bit works: Quantization explained.
• Size the machine before you buy: Edge AI hardware guide 2026.
• Decide whether the box pays for itself: Cloud vs local AI cost benchmarks.

Work with us

We run this measurement for clients on their own hardware before recommending a model, and we publish the numbers we get. If you want a second opinion on a file choice or a machine, get in touch or read how our consulting works.