By the end of this post you will know which of the two most common local model servers fits your team, and at what number of people the answer flips. You can decide on the machine you already have: both servers are free, open source, and run the same model weights. We put the same request through each of them, on two identical boxes, with one to ten people pressing Enter at once.
What you need
- A machine that already runs a 7B model. Our Ollama install guide gets you there on a laptop.
- Docker, if you want to try vLLM.
- Python 3 and our script
scripts/bench/openwebui_concurrency.pyfrom the repository, to fire the same requests at both. - About an hour, most of it downloads: the 16-bit model is 15 GB per server.
Step 1: install Ollama and serve the model
Ollama is one command to install and one to pull a model. We pulled the 16-bit tag on purpose, so that both servers run the same weights at the same precision:
curl -fsSL https://ollama.com/install.sh | sh
ollama pull qwen2.5:7b-instruct-fp16
ollama ps # after the first request: 16.4 GB loaded at a 32k context on our boxThe default tag, qwen2.5:7b, is 4-bit: a third of the size and faster. We did not use it here because then we would be comparing a quantisation, not a server. If you only ever run Ollama, use the 4-bit tag; our GGUF guide explains the trade.
Step 2: run vLLM in Docker
vLLM wants the original Hugging Face weights and a GPU it can keep them on. This is the compose file from our second box, trimmed to what matters:
services:
vllm:
image: nvcr.io/nvidia/vllm:26.04-py3
ports: ["8200:8200"]
volumes: ["${HOME}/.cache/huggingface:/root/.cache/huggingface"]
ipc: host
command: >
vllm serve Qwen/Qwen2.5-7B-Instruct
--host 0.0.0.0 --port 8200
--gpu-memory-utilization 0.16
--max-model-len 32768
--served-model-name default
deploy:
resources:
reservations:
devices: [{driver: nvidia, count: all, capabilities: [gpu]}]--gpu-memory-utilization 0.16 is the share of the box’s memory vLLM may take for weights and cache: 16 % of 128 GB here, because the box serves other models too. On a single-purpose GPU you would set it higher. The first start downloads 15 GB and compiles kernels; budget minutes, not seconds. Both servers then speak the same OpenAI-style API, which is why one script can test both.
Step 3: fire the same requests at both
The request is an office task: a client email about an invoice, 384 tokens in, a 250-token reply out, temperature 0.7, streaming. The script sends N of them at the same instant, three times per level, and records when the first word arrives and when the answer ends.
python3 scripts/bench/openwebui_concurrency.py --base-url http://localhost:11434/v1 \
--model qwen2.5:7b-instruct-fp16 --levels 1,2,5,10 --reps 3 --max-tokens 250 \
--out docs/measurements/ollama-concurrency-prime-2026-11-02.json
python3 scripts/bench/openwebui_concurrency.py --base-url http://<second-box>:8200/v1 \
--model default --levels 1,2,5,10 --reps 3 --max-tokens 250 \
--out docs/measurements/vllm-concurrency-spark2-2026-11-02.jsonWhat we measured
Measured 2026-11-02 on two NVIDIA DGX Sparks (GB10, 128 GB unified memory each): Ollama 0.30.10 on the first, vLLM 0.19.0 (NVIDIA container 26.04) on the second, so the servers never shared a GPU. Same weights (Qwen2.5-7B-Instruct, 16-bit), same prompt, medians over three repetitions, “max” is the slowest single request. Raw files: ollama-concurrency-prime-2026-11-02.json and vllm-concurrency-spark2-2026-11-02.json.
Ollama (qwen2.5:7b-instruct-fp16, 16.4 GB loaded, 32k context, OLLAMA_NUM_PARALLEL left at its default):
| Users at once | First word, median (max) | Full answer, median (max) | tok/s per user | tok/s total |
|---|---|---|---|---|
| 1 | 0.21 s (0.23 s) | 15.9 s (16.9 s) | 16.0 | 15.8 |
| 2 | 8.0 s (16.0 s) | 23.3 s (31.6 s) | 15.9 | 15.7 |
| 5 | 31.2 s (62.6 s) | 46.5 s (79.2 s) | 16.0 | 15.9 |
| 10 | 71.7 s (146.4 s) | 86.6 s (162.9 s) | 15.8 | 15.7 |
vLLM (Qwen/Qwen2.5-7B-Instruct BF16, --gpu-memory-utilization 0.16, --max-model-len 32768):
| Users at once | First word, median (max) | Full answer, median (max) | tok/s per user | tok/s total |
|---|---|---|---|---|
| 1 | 0.08 s (0.08 s) | 18.2 s (18.3 s) | 13.8 | 13.7 |
| 2 | 0.08 s (0.09 s) | 14.8 s (15.4 s) | 16.9 | 32.1 |
| 5 | 0.10 s (0.18 s) | 14.9 s (15.7 s) | 16.6 | 80.1 |
| 10 | 0.17 s (0.18 s) | 15.1 s (15.2 s) | 16.6 | 158.6 |
The vLLM rows reproduce our run of 2026-10-08 on the same box within 2 %, which is the kind of boring result you want from a benchmark.
Three things the tables say.
Alone, Ollama is faster. One person got 16.0 tokens per second from Ollama and 13.8 from vLLM, and the full answer two seconds sooner. If that is your office, you are done: install Ollama and stop reading.
The crossover is at two. With two people, Ollama’s second person waited 16 seconds to see a word, because the first request had to finish. vLLM gave both their first word in 0.08 seconds and both their answers in 15. Look at the per-user column: vLLM’s users got faster at two than at one, because its batching uses the GPU better with more than one stream.
At ten, it is a queue against a counter. The ten Ollama requests saw their first word at 0.2, 15.7, 31.6, 47.1, 63.0, 79.5, 94.9, 109.3, 124.8 and 140.2 seconds: one at a time, each taking about 15.6 seconds, total output stuck at one user’s 15.7 tokens per second. vLLM served all ten together, first word in under 0.2 seconds, every answer in about 15 seconds, 158.6 tokens per second out of the box.
What the spec sheet won’t tell you
- Ollama’s queue is a setting, not a law. It runs a fixed number of parallel slots and queues the rest;
OLLAMA_NUM_PARALLELraises the count, and each slot costs memory for its own context cache. We measured the default our installation ships with and did not measure other values, so this post makes no claim about them. We measured the other values later: Ollama with several users, measured. - Precision was equal here; by default it is not. Ollama’s usual tags are 4-bit and smaller and faster than the 16-bit file we used. vLLM runs the original weights unless you pass it a quantised model.
- vLLM pins its memory at start; Ollama borrows it. vLLM takes its share of the GPU when the container starts and keeps it. Ollama loads on the first request and unloads after
OLLAMA_KEEP_ALIVE(10 minutes on our box). On a shared machine that difference decides who else can run. - Both boxes were busy. During the Ollama run another client loaded an 8B model on the same box, which is why one 2-user request in the raw file took 2 min 34 s; the medians above are unaffected. The vLLM box kept a 21 GB chat model resident throughout. That is normal on our stack and probably on yours.
Verdict
- Ollama. One person at a time, or a batch job that runs one request after another. Faster per token, one command, nothing to configure.
- vLLM. Two or more people who may ask at the same time, or any chat page where a 70-second pause is a complaint. Ten people got their answer in 15 seconds each.
- Neither yet. If you have not run a 7B model on the machine you own, do that first with Ollama and come back when someone has to wait.
When not to do this
We measured one model, one prompt length and 250-token answers. Paste a 30-page contract and every request carries thousands of tokens; the first word takes longer and memory per user grows on both servers. We have not measured that case. We also used two boxes, which is why the losing server cost nothing: if your office has one GPU and both servers must share it, run the script again before deciding. And if the office asks three questions a day, the cloud model you already pay for is fine.
Next steps
- Put the server behind a login page: Open WebUI for a 10-person office.
- Size the memory before you pick a model: How much VRAM do I need?.
- Lock it down: the 10-minute Ollama security checklist. To get your own version of these tables on your own machine, start with Benchmark a local model in 30 minutes.
Work with us
We size the model and the machine by measuring, not by guessing. If you want to see your own task running on real hardware, book a 15-minute call or see how we work in consulting.