By the end of this post you will know what each context size costs in memory, why the same model can take 5 GB on one server and 39 GB on another, and where to set num_ctx so a load does not fail. It works on any machine that runs Ollama, a laptop included. We learned it on a box with 128 GB, which turned out not to be enough.
The day our models stopped loading
On 2026-10-11 every model load on our second DGX Spark hung or was killed for lack of memory, unless the model carried its own context size. The box has 128 GB of memory.
The log said why. Ollama 0.34 picks a default context from the memory it sees, and on that box it chose 262,144 tokens (default_num_ctx=262144). The server also runs 8 parallel slots, and each slot reserves its own context. llama3.1:8b, a 4.9 GB file, had reached 36 GB of resident memory when we stopped it.
Two models kept working. One had 32k written into its Modelfile; the other was our news-desk judge, which sends num_ctx with every request. That gave us the fix before we opened any documentation. The same day we wrote a num_ctx into every model on that box (16384 for almost all of them), and later set OLLAMA_CONTEXT_LENGTH=32768 in the server’s unit.
Then we measured what each choice costs, so you can skip the bad afternoon.
What you need
- Ollama on any machine. Our install guide gets you there on a laptop.
- A model you already use.
curl, andsudoonly for step 4.- Optional: the same table for your machine. Our script
scripts/bench/ollama_num_ctx.pyis in our internal repository and not public yet; loading each size and readingollama ps(step 1) gives you the same numbers by hand.
Step 1: see what context you are paying for
ollama ps
# NAME SIZE PROCESSOR CONTEXT
# llama3.1:8b 9.2 GB 100% GPU 32768
ollama show llama3.1:8b --parameters
# num_ctx 32768SIZE is what the loaded model holds in memory. CONTEXT is the context of one slot, while SIZE pays for all of them. If show lists no num_ctx, the model takes the server default, and that default depends on your version and your memory. On a Linux install, look at the server’s settings and at what it decided:
systemctl show ollama --property=Environment
journalctl -u ollama | grep -i num_ctx | tail -3Step 2: set it per request
Any call to Ollama’s own API can carry the size:
curl http://localhost:11434/api/chat -d '{
"model": "llama3.1:8b",
"messages": [{"role": "user", "content": "Summarise this email: ..."}],
"options": {"num_ctx": 8192}
}'Keep the value the same across a client. A request with a different size makes Ollama load the model again.
The OpenAI-style endpoint (/v1/chat/completions) has no field for it. Ollama’s compatibility page says to use a Modelfile instead, which is step 3.
Step 3: write it into the model
This is the fix that worked for us. A Modelfile of two lines gives the model a context of its own:
cat > Modelfile <<'EOF'
FROM qwen3.6:35b
PARAMETER num_ctx 16384
EOF
ollama create qwen3.6-16k -f Modelfile
ollama show qwen3.6-16k --parameters | grep num_ctxEvery client that asks for qwen3.6-16k now gets 16k, including those on the /v1 endpoint. On our second box we overwrote the original names instead, so no client had to change:
curl http://localhost:11434/api/create -d '{"model": "qwen3.6:35b", "from": "qwen3.6:35b", "parameters": {"num_ctx": 16384}}'A model you pull later arrives without that line. Add it the same day.
Step 4: set a server default
The default catches any model you forgot. On a systemd install:
sudo systemctl edit ollama
# [Service]
# Environment="OLLAMA_CONTEXT_LENGTH=16384"
# Environment="OLLAMA_NUM_PARALLEL=4"
sudo systemctl restart ollamaThe order is: the request’s num_ctx first, then the model’s, then the server default.
What we measured
Measured 2026-11-02 on two NVIDIA DGX Sparks (GB10, 128 GB unified memory each). First box: Ollama 0.30.10, two temporary instances with 1 and 4 parallel slots. Second box: our production Ollama 0.34.0, 8 slots, another model resident throughout. For each row the script unloads the model, loads it with one 8-token request that carries num_ctx, reads size from /api/ps, and unloads it again. All models are Q4_K_M. Raw file: ollama-num-ctx-2026-11-02.json.
python3 scripts/bench/ollama_num_ctx.py --ollama http://127.0.0.1:11437 --num-parallel 1 \
--models llama3.1:8b,gemma4:26b --ctx 4096,32768,131072 --append \
--out docs/measurements/ollama-num-ctx-2026-11-02.json
python3 scripts/bench/ollama_num_ctx.py --ollama http://127.0.0.1:11438 --num-parallel 4 \
--models llama3.1:8b,gemma4:26b --ctx 4096,32768 --append \
--out docs/measurements/ollama-num-ctx-2026-11-02.json
python3 scripts/bench/ollama_num_ctx.py --ollama http://<second-box>:11434 --num-parallel 8 \
--mem-host <second-box> --models llama3.1:8b,qwen3.6:35b --ctx 4096,16384,32768 --append \
--out docs/measurements/ollama-num-ctx-2026-11-02.jsonOne slot (first box, OLLAMA_NUM_PARALLEL=1):
| num_ctx | llama3.1:8b (4.9 GB file) | gemma4:26b (18.0 GB file) |
|---|---|---|
| 4,096 | 5.27 GB | 17.20 GB |
| 32,768 | 9.21 GB | 17.47 GB |
| 131,072 | 22.79 GB | 18.36 GB |
llama3.1:8b across slots (reserved tokens = slots × num_ctx):
| Box, slots | num_ctx | Reserved tokens | Loaded size |
|---|---|---|---|
| first, 1 | 4,096 | 4,096 | 5.27 GB |
| first, 4 | 4,096 | 16,384 | 7.00 GB |
| second, 8 | 4,096 | 32,768 | 9.15 GB |
| first, 1 | 32,768 | 32,768 | 9.21 GB |
| second, 8 | 16,384 | 131,072 | 22.32 GB |
| first, 4 | 32,768 | 131,072 | 22.38 GB |
| first, 1 | 131,072 | 131,072 | 22.79 GB |
| second, 8 | 32,768 | 262,144 | 39.27 GB |
The two that barely grow: gemma4:26b with 4 slots loaded at 18.67 GB (4k) and 19.84 GB (32k). qwen3.6:35b (23.9 GB file) with 8 slots loaded at 22.72 GB (4k), 23.10 GB (16k) and 23.47 GB (32k).
Three things the tables say.
Slots times context is the number that counts. 32,768 reserved tokens cost about 9.2 GB as one slot of 32k or as eight slots of 4k. 131,072 cost 22.3 to 22.8 GB in three layouts, on two boxes and two Ollama versions. Ollama’s FAQ says memory scales with OLLAMA_NUM_PARALLEL × OLLAMA_CONTEXT_LENGTH (read 2026-11-02). Now you have the numbers.
On a classic model each reserved token costs about 0.14 MB. From 32,768 to 131,072 tokens, llama3.1:8b grew 13.58 GB: 0.138 MB per token. The formula in How much VRAM do I need? gives 0.131 MB for this model, close enough to plan with.
Not every model pays like that. gemma4:26b gained 1.16 GB from 4k to 128k; Llama gained 17.52 GB over the same range. We did not measure why. Our reading is that both newer families keep most layers’ cache at a fixed size, so only a few layers pay per token. Treat that as a hypothesis and measure your own model.
What the spec sheet won’t tell you
- The size is reserved at load, not when you use it. Our loads were 8-token requests and already showed the full size. A long context you never fill still costs memory from the first message.
- The documentation and your build can disagree. The FAQ says the default context is 4,096 tokens (read 2026-11-02). Our 0.34.0 chose 262,144 on a 128 GB box. Trust the log line from step 1.
- A server default is a safety net, not a fit. With
OLLAMA_CONTEXT_LENGTH=32768and 8 slots, a Llama 3.1 8B without its own value loads at 39.27 GB. That is why every model on our second box also carries 16k (22.32 GB). - We did not re-run the 262,144 case. We know which loads failed, not which buffer failed first in each one. Breaking a production box once was enough.
Verdict
- Write it into every model.
PARAMETER num_ctxsized to the longest real task, not to the number on the model card. - Then set a server default, lower than feels comfortable, for the model you will forget.
- Fewer slots before more memory. Four slots at 32k cost what one slot at 128k costs. Pick the one your users need. What each slot buys in speed: num_parallel and keep_alive, measured.
When not to do this
If one person runs one model on a laptop, check ollama ps once; your default may already be fine. If your task needs the long context (a full contract, a long transcript), a small num_ctx is worse than a slow load, because Ollama cuts the start of the prompt and only says so in its log. And if that long job happens twice a month, a hosted model with a large window may cost less than the memory.
Next steps
- Turn these sizes into a buying decision: How much VRAM do I need?.
- Before you set
num_ctx, count how many tokens your longest task takes with the tokenizer. - When several people ask at once, slots matter more: Ollama vs vLLM for 1 to 10 users.
- Put the model behind accounts for the office: Open WebUI for a 10-person office.
- Get told when a load fails instead of finding out from a colleague: keep a local AI server running.
Work with us
We size the model and the machine by measuring, not by guessing. If you want to see your own task running on real hardware, book a free 15-minute call or see how we work in consulting.