In 30 minutes you will know how many tokens per second your own machine produces with three local models, and which of them is worth keeping. No new hardware, no account, no API key.
What you need
- Hardware: the laptop or desktop you already have. Rough guide for memory, taken from the loaded sizes in our table below plus room for the operating system:
- 8 GB RAM: 3B to 4B models.
- 16 GB RAM: 7B to 8B models.
- 32 GB or more: 14B to 26B models.
- Software: Ollama (free, open source) and a terminal.
- Time: 30 minutes, most of it downloads.
On a Mac with Apple Silicon the GPU shares system memory, so the RAM rule above is the whole story. On Windows or Linux with a separate GPU, what matters is how much of the model fits in the card’s VRAM.
Step 1: Install Ollama
Ollama runs the model and hides the llama.cpp plumbing behind one command. On macOS or Windows, use the installer from ollama.com. On Linux:
curl -fsSL https://ollama.com/install.sh | sh
ollama --versionExpected: ollama version 0.x.x.
If it fails: command not found means the install did not finish or your terminal has not picked up the new PATH. Open a new terminal. If the service will not start, something else is using port 11434.
Step 2: Pull three models of different sizes
Pick a small, a medium and a large one. Three models of the same size tell you nothing; three sizes show you where your machine stops coping.
ollama pull llama3.1:8b
ollama pull qwen2.5-coder:7b
ollama pull gemma4:26bOn 8 GB of RAM, swap gemma4:26b for a 3B model such as llama3.2:3b.
If it fails: a dropped download resumes if you run the same pull again. “Insufficient space” means you need about 28 GB free (4.9 + 4.7 + 18 GB for these three).
Use the same prompt for every run so the only thing changing is the model:
“Analyze the following text for key entities and summarize the main action in one sentence: The quick brown fox jumps over the lazy dog near the riverbank in Valencia.”
Step 3: Measure, twice per model
The first run loads the model from disk, so its numbers include loading time. Throw it away and keep the second.
ollama run llama3.1:8b --verbosePaste the prompt. After the answer, Ollama prints:
| Line | What it tells you |
|---|---|
eval rate | Generation speed, tokens per second. The number you care about. |
prompt eval rate | How fast it reads your prompt. |
load duration | Time to get the model into memory (near zero on the second run). |
total duration | Everything together. |
While the model is still loaded, run ollama ps in a second terminal and write down the size column.
Prefer something you can script? The API returns the same counters as JSON:
curl http://localhost:11434/api/generate -d '{
"model": "llama3.1:8b",
"prompt": "Analyze the following text for key entities and summarize the main action in one sentence: The quick brown fox jumps over the lazy dog near the riverbank in Valencia.",
"stream": false
}'Divide eval_count by eval_duration (nanoseconds) and multiply by 10⁹ for tokens per second.
If it hangs: the model does not fit in memory and the machine is swapping to disk. Stop it with Ctrl+C and try a smaller model or a smaller file; our GGUF quantization guide shows how to pick one.
Step 4 (optional): a second opinion with llama-bench
To check the numbers are not an Ollama quirk, llama-bench from llama.cpp measures reading and writing speed separately at fixed lengths:
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build
cmake --build build --config Release --target llama-bench
./build/bin/llama-bench -m path/to/your/model.gguf -p 512 -n 512It prints one row for prompt processing and one for token generation. llama.cpp builds with CMake only (its old Makefile now just prints an error), so a failed build usually means CMake or a C++ compiler is missing.
What we measured
Our reference runs, so you have something to compare against. Fill in the last row with yours.
| Model | Loaded size | Generation tok/s | Prompt tok/s | Time to first token (s) | Wall time, 200 tokens (s) |
|---|---|---|---|---|---|
| llama3.1:8b | 9.2 GB | 40.2 | 3342.4 | 0.37 | 5.3 |
| qwen2.5-coder:7b | 6.6 GB | 42.1 | 3162.0 | 0.33 | 5.1 |
| gemma4:26b | 17.5 GB | 62.0 | 1864.3 | 0.69 | 3.9 |
| Yours (second run) |
Measured 2026-10-17 on our NVIDIA DGX Spark (GB10, 128 GB unified memory), Ollama 0.30.10, second run, 200-token limit, temperature 0. Our runs used a longer prompt (278 words) than the one above, so compare your generation column with ours; prompt speed depends on prompt length. The script is scripts/bench/local_model_benchmark.py and every run, cold first runs included, is in docs/measurements/local-model-benchmark-2026-10-17.json in our repo.
Two traps we hit while measuring. Send the identical prompt twice and Ollama reuses the first run’s work: our second run then “read” 333 tokens in 0.03 seconds, over 10,000 tok/s, a speed no new prompt will ever see. Change the first line of the prompt between runs (we added “Request 2 of 3.”). And gemma4 is a thinking model: with thinking on, it reasons before the first visible word. We turned it off ("think": false in the API) so all three models wrote 200 visible tokens.
What the spec sheet won’t tell you
- Bigger is not always slower. The 26B model generated about 50% faster than both the 7B and the 8B model on our box. gemma4:26b is a mixture-of-experts model: it has 128 experts and uses 8 of them per token, so each token touches a fraction of its 26B parameters. It still needs the memory for all of them. Parameter count alone does not predict speed; architecture and the file you download matter as much. That is exactly why you measure.
- Chat needs less than you think. Anything above roughly reading speed feels instant in a chat window. Batch jobs (summarising 500 documents overnight) are where tokens per second turn into hours.
- Memory is the hard wall. A model that does not fit does not run slowly; it barely runs at all.
When not to bother
If the job is one long document a month, the cloud model you already pay for is fine. Local wins when the data cannot leave your building, when the volume is high, or when you want the same answer every time without a price change.
Next steps
- Pick the right file size for your RAM: GGUF quantization explained.
- See what each budget buys: hardware by budget.
- Work out the memory before you download: VRAM calculator.
Work with us
We size the model and the machine by measuring, not by guessing. If you want to see your own task running on real hardware, book a 15-minute call or see how we work in consulting.