View all articles
OllamavLLMBenchmarksSelf-hostedLocal AI

Ollama vs vLLM for 1 to 10 Users: Same Model, Same Box, Measured

JG
Jacobo González Jaspe
|
Abstract illustration: glass columns of different heights; the tallest glows amber. Benchmark del mes
Illustration generated with AI on our own machine.

By the end of this post you will know which of the two most common local model servers fits your team, and at what number of people the answer flips. You can decide on the machine you already have: both servers are free, open source, and run the same model weights. We put the same request through each of them, on two identical boxes, with one to ten people pressing Enter at once.

What you need

  • A machine that already runs a 7B model. Our Ollama install guide gets you there on a laptop.
  • Docker, if you want to try vLLM.
  • Python 3 and our script scripts/bench/openwebui_concurrency.py from the repository, to fire the same requests at both.
  • About an hour, most of it downloads: the 16-bit model is 15 GB per server.

Step 1: install Ollama and serve the model

Ollama is one command to install and one to pull a model. We pulled the 16-bit tag on purpose, so that both servers run the same weights at the same precision:

bash
curl -fsSL https://ollama.com/install.sh | sh
ollama pull qwen2.5:7b-instruct-fp16
ollama ps   # after the first request: 16.4 GB loaded at a 32k context on our box

The default tag, qwen2.5:7b, is 4-bit: a third of the size and faster. We did not use it here because then we would be comparing a quantisation, not a server. If you only ever run Ollama, use the 4-bit tag; our GGUF guide explains the trade.

Step 2: run vLLM in Docker

vLLM wants the original Hugging Face weights and a GPU it can keep them on. This is the compose file from our second box, trimmed to what matters:

YAML
services:
  vllm:
    image: nvcr.io/nvidia/vllm:26.04-py3
    ports: ["8200:8200"]
    volumes: ["${HOME}/.cache/huggingface:/root/.cache/huggingface"]
    ipc: host
    command: >
      vllm serve Qwen/Qwen2.5-7B-Instruct
      --host 0.0.0.0 --port 8200
      --gpu-memory-utilization 0.16
      --max-model-len 32768
      --served-model-name default
    deploy:
      resources:
        reservations:
          devices: [{driver: nvidia, count: all, capabilities: [gpu]}]

--gpu-memory-utilization 0.16 is the share of the box’s memory vLLM may take for weights and cache: 16 % of 128 GB here, because the box serves other models too. On a single-purpose GPU you would set it higher. The first start downloads 15 GB and compiles kernels; budget minutes, not seconds. Both servers then speak the same OpenAI-style API, which is why one script can test both.

Step 3: fire the same requests at both

The request is an office task: a client email about an invoice, 384 tokens in, a 250-token reply out, temperature 0.7, streaming. The script sends N of them at the same instant, three times per level, and records when the first word arrives and when the answer ends.

bash
python3 scripts/bench/openwebui_concurrency.py --base-url http://localhost:11434/v1 \
  --model qwen2.5:7b-instruct-fp16 --levels 1,2,5,10 --reps 3 --max-tokens 250 \
  --out docs/measurements/ollama-concurrency-prime-2026-11-02.json

python3 scripts/bench/openwebui_concurrency.py --base-url http://<second-box>:8200/v1 \
  --model default --levels 1,2,5,10 --reps 3 --max-tokens 250 \
  --out docs/measurements/vllm-concurrency-spark2-2026-11-02.json

What we measured

Measured 2026-11-02 on two NVIDIA DGX Sparks (GB10, 128 GB unified memory each): Ollama 0.30.10 on the first, vLLM 0.19.0 (NVIDIA container 26.04) on the second, so the servers never shared a GPU. Same weights (Qwen2.5-7B-Instruct, 16-bit), same prompt, medians over three repetitions, “max” is the slowest single request. Raw files: ollama-concurrency-prime-2026-11-02.json and vllm-concurrency-spark2-2026-11-02.json.

Ollama (qwen2.5:7b-instruct-fp16, 16.4 GB loaded, 32k context, OLLAMA_NUM_PARALLEL left at its default):

Users at onceFirst word, median (max)Full answer, median (max)tok/s per usertok/s total
10.21 s (0.23 s)15.9 s (16.9 s)16.015.8
28.0 s (16.0 s)23.3 s (31.6 s)15.915.7
531.2 s (62.6 s)46.5 s (79.2 s)16.015.9
1071.7 s (146.4 s)86.6 s (162.9 s)15.815.7

vLLM (Qwen/Qwen2.5-7B-Instruct BF16, --gpu-memory-utilization 0.16, --max-model-len 32768):

Users at onceFirst word, median (max)Full answer, median (max)tok/s per usertok/s total
10.08 s (0.08 s)18.2 s (18.3 s)13.813.7
20.08 s (0.09 s)14.8 s (15.4 s)16.932.1
50.10 s (0.18 s)14.9 s (15.7 s)16.680.1
100.17 s (0.18 s)15.1 s (15.2 s)16.6158.6

The vLLM rows reproduce our run of 2026-10-08 on the same box within 2 %, which is the kind of boring result you want from a benchmark.

Three things the tables say.

Alone, Ollama is faster. One person got 16.0 tokens per second from Ollama and 13.8 from vLLM, and the full answer two seconds sooner. If that is your office, you are done: install Ollama and stop reading.

The crossover is at two. With two people, Ollama’s second person waited 16 seconds to see a word, because the first request had to finish. vLLM gave both their first word in 0.08 seconds and both their answers in 15. Look at the per-user column: vLLM’s users got faster at two than at one, because its batching uses the GPU better with more than one stream.

At ten, it is a queue against a counter. The ten Ollama requests saw their first word at 0.2, 15.7, 31.6, 47.1, 63.0, 79.5, 94.9, 109.3, 124.8 and 140.2 seconds: one at a time, each taking about 15.6 seconds, total output stuck at one user’s 15.7 tokens per second. vLLM served all ten together, first word in under 0.2 seconds, every answer in about 15 seconds, 158.6 tokens per second out of the box.

What the spec sheet won’t tell you

  • Ollama’s queue is a setting, not a law. It runs a fixed number of parallel slots and queues the rest; OLLAMA_NUM_PARALLEL raises the count, and each slot costs memory for its own context cache. We measured the default our installation ships with and did not measure other values, so this post makes no claim about them. We measured the other values later: Ollama with several users, measured.
  • Precision was equal here; by default it is not. Ollama’s usual tags are 4-bit and smaller and faster than the 16-bit file we used. vLLM runs the original weights unless you pass it a quantised model.
  • vLLM pins its memory at start; Ollama borrows it. vLLM takes its share of the GPU when the container starts and keeps it. Ollama loads on the first request and unloads after OLLAMA_KEEP_ALIVE (10 minutes on our box). On a shared machine that difference decides who else can run.
  • Both boxes were busy. During the Ollama run another client loaded an 8B model on the same box, which is why one 2-user request in the raw file took 2 min 34 s; the medians above are unaffected. The vLLM box kept a 21 GB chat model resident throughout. That is normal on our stack and probably on yours.

Verdict

  • Ollama. One person at a time, or a batch job that runs one request after another. Faster per token, one command, nothing to configure.
  • vLLM. Two or more people who may ask at the same time, or any chat page where a 70-second pause is a complaint. Ten people got their answer in 15 seconds each.
  • Neither yet. If you have not run a 7B model on the machine you own, do that first with Ollama and come back when someone has to wait.

When not to do this

We measured one model, one prompt length and 250-token answers. Paste a 30-page contract and every request carries thousands of tokens; the first word takes longer and memory per user grows on both servers. We have not measured that case. We also used two boxes, which is why the losing server cost nothing: if your office has one GPU and both servers must share it, run the script again before deciding. And if the office asks three questions a day, the cloud model you already pay for is fine.

Next steps

Work with us

We size the model and the machine by measuring, not by guessing. If you want to see your own task running on real hardware, book a 15-minute call or see how we work in consulting.

Diagram
Share: LinkedIn X
Veredicto semanal

Get new guides before anyone else

Subscribe and we tell you when new guides, templates and workflows go up. One email a week, no spam.

Already published: 102 guides and 25 templates. All free, no signup.

Bonus: the local-AI starter pack PDF when you subscribe
Once a week No spam Unsubscribe anytime

See what you get