View all articles
Model ReviewLlama 3.3Local AIHardwareQuantization

Llama 3.3 70B locally: the memory sum, hardware and speed

JG
Jacobo González Jaspe
|

Reviewed:

Abstract illustration: a network of white nodes joined by amber lines, like a constellation. Models
Illustration generated with AI on our own machine.

By the end of this guide you will know whether Llama 3.3 70B fits the machine you have, what you would need if it does not, what speed to expect and whether a smaller model would serve you better. A calculator is enough to follow it. A 16 GB laptop is not enough to run it, and nothing below changes that.

One thing first. We have not run Llama 3.3 70B. Our machines are shared, and a 43 GB load on them was a risk we chose not to take. So this is not a test report. The model’s facts come from Meta and the file repositories, the memory figures from a sum checked against real loads of other models, and the 70B speeds from other people, named. What we measured ourselves is the largest models we do run.

The model’s facts

From Meta’s model card, read on 2026-11-15:

FactValue
Released6 December 2024
TypeDense text model: text in, text and code out. No images.
Parameters70 billion (the 16-bit file is 141.12 GB: 70.6 billion at two bytes each)
Context128k tokens (131,072 in its config.json)
Languages supportedEnglish, German, French, Italian, Portuguese, Hindi, Spanish, Thai
Knowledge cutoffDecember 2023
LicenceLlama 3.3 Community License

Two things follow for a Spanish office. Spanish is on the supported list; for languages outside it, the card discourages use without fine-tuning and controls of your own. And the model knows nothing after 2023, so current rules or prices have to come from your documents.

Meta’s table on the card gives it 68.9 on MMLU Pro against 48.3 for Llama 3.1 8B, and 92.1 on instruction following (IFEval) against 80.4. Those are Meta’s numbers under Meta’s conditions.

What “open” means here

This section is information, not legal advice. We read the licence and the acceptable use policy on 2026-11-15; read them yourself before a client depends on the model.

The weights are free to download and you may use them commercially. That is the “open” part. But the model card calls the licence “a custom commercial license”: it is Meta’s own agreement, not Apache or MIT. What it asks:

  • If you redistribute the model or a product containing it (section 1.b): pass on a copy of the agreement, show “Built with Llama” somewhere prominent, and keep the attribution notice.
  • If you use it to train or improve another model that you distribute: that model’s name must begin with “Llama”.
  • Above 700 million monthly active users (section 2) you need a separate licence from Meta.
  • The acceptable use policy binds you (section 1.b.iv), and Meta may end the licence if you breach the agreement (section 6). California law governs it (section 7).

From the use policy, an office should know these: no unlicensed practice of a profession (1.d), no military or critical-infrastructure use (2.a and 2.d), no presenting the output as written by a person (3.e), and a duty to tell end users about known dangers of your system (4).

The policy also withholds the licence from people and companies based in the EU for any multimodal models. Llama 3.3 70B is text only, so as we read it that clause does not reach this model.

Meta’s repository on Hugging Face is gated: you accept the licence and share contact details before downloading.

File sizes by quantisation

Sizes from bartowski’s GGUF repository, read from its file list on 2026-11-15. The quality notes are that repository’s, in our words. Tags are from Ollama’s list, same day.

QuantisationFileOllama tagThe repository’s note
Q8_074.98 GBllama3.3:70b-instruct-q8_0Highest quality, usually more than needed
Q6_K57.89 GBllama3.3:70b-instruct-q6_KNear perfect, recommended
Q5_K_M49.95 GBllama3.3:70b-instruct-q5_K_MHigh quality, recommended
Q4_K_M42.52 GBllama3.3:70b (default)Good quality, the default for most uses
IQ4_XS37.90 GBnoneDecent, recommended
Q2_K26.38 GBllama3.3:70b-instruct-q2_KVery low quality but usable
IQ2_XXS19.10 GBnoneVery low quality

Ollama rounds the default to 43 GB. How to choose between these labels is in our guide to picking the right GGUF file. We have no quality measurement of our own for any of them.

The memory sum

The file is not the whole bill. The method is the one in our VRAM guide, where the sum matched 13 real loads of smaller models to the megabyte (load-log receipt):

text
memory = weights (the file) + context cache × parallel slots + compute buffers

Step 1: read the three values

The cache depends on layers, KV heads and head dimension, all in the model’s config.json. Meta’s repository is gated, so this reads an ungated copy:

bash
curl -sL https://huggingface.co/unsloth/Llama-3.3-70B-Instruct/raw/main/config.json | python3 -c "
import sys, json
c = json.load(sys.stdin)
for k in ('num_hidden_layers', 'num_key_value_heads', 'head_dim', 'hidden_size', 'vocab_size'): print(k, '=', c[k])"

What it printed for us on 2026-11-15:

text
num_hidden_layers = 80
num_key_value_heads = 8
head_dim = 128
hidden_size = 8192
vocab_size = 128256

Step 2: the cache, by context

Each token stores a K and a V vector per layer, at 16 bits: 80 × 8 × 128 × 2 × 2 = 327,680 bytes per token. That is two and a half times what Llama 3.1 8B needs, and the file’s quantisation does not change it.

Python
FILE_GB = 42.52                                    # Q4_K_M file
per_token = 80 * 8 * 128 * 2 * 2                   # bytes
for ctx, slots in [(4096, 1), (8192, 1), (16384, 1), (32768, 1), (131072, 1), (8192, 8)]:
    cache = per_token * ctx * slots / 1e9
    print(f"{ctx:>7} x {slots}: cache {cache:5.2f} GB, weights + cache {FILE_GB + cache:6.2f} GB")

We ran it (it loads nothing):

text
   4096 x 1: cache  1.34 GB, weights + cache  43.86 GB
   8192 x 1: cache  2.68 GB, weights + cache  45.20 GB
  16384 x 1: cache  5.37 GB, weights + cache  47.89 GB
  32768 x 1: cache 10.74 GB, weights + cache  53.26 GB
 131072 x 1: cache 42.95 GB, weights + cache  85.47 GB
   8192 x 8: cache 21.47 GB, weights + cache  63.99 GB

At the full 128k the cache is as large as the model.

Step 3: the slots

The last line is the same model serving eight people at once. Ollama’s FAQ says required memory scales with OLLAMA_NUM_PARALLEL times the context length; the default is 1. Each extra user at 8k costs this model 2.68 GB. Our second machine runs eight slots, and there a 24B model went from 19.79 GB to 25.64 GB loaded just by doubling the context from 4k to 8k (4k, 8k).

Step 4: buffers, and what ollama ps leaves out

llama.cpp also reserves working memory: 112 to 912 MiB in our logs, depending on batch size and slots. We allow 1 GB for a 70B. That is an allowance, not a measurement of this model.

ollama ps may then show less than the file size. It reports the GPU buffers, and the token embedding table stays on the host side. In our Llama 3.1 8B load that table is 281.81 MiB: its 128,256 × 4,096 values at 4.5 bits each. The 70B has 128,256 × 8,192. Stored the same way, that is 0.59 GB missing from ollama ps and still in memory. We have not checked it on a real load.

What fits in each machine

Planning sum for Q4_K_M with one slot (file + cache + 1 GB): 44.86 GB at 4k, 46.20 GB at 8k, 54.26 GB at 32k and 86.47 GB at 128k. As in the VRAM guide, a card keeps 0.5 GB back and unified memory above 32 GB keeps 6 GB for the system. “Tight” means over 85% of what is left.

Machine memoryUsable4k8k32k128k
One 24 GB card23.5 GBnononono
One 32 GB card31.5 GBnononono
Two 24 GB cards47 GBtighttightnono
One 48 GB card47.5 GBtighttightnono
64 GB unified58 GByesyestightno
96 GB unified90 GByesyesyestight
128 GB unified122 GByesyesyesyes

How to read it:

  • 24 and 32 GB. Only IQ2_XXS fits in 24 GB (21.44 GB at 4k) and Q2_K in 32 GB (30.06 GB at 8k, tight). The repository rates both very low quality. A good 24B at Q4 is a better use of those cards.
  • 48 GB. Q4_K_M works up to 8k with almost nothing spare; at 16k the sum is 48.89 GB and it is out. IQ4_XS buys room: 44.27 GB at 16k. Two cards are not one pool, since whole layers go to one or the other, so expect less headroom. We have not run a split load.
  • 64 GB unified. The first size where it is comfortable. Q5_K_M also fits up to 16k (56.32 GB, tight).
  • 128 GB unified. Q8_0 at 32k is 86.72 GB. But “128 GB” is not 128 free: our own machines report 121.7 GB in total, and with their usual services had 65.4 to 75.1 GB available before the loads in our receipts (first machine, second). That is why we did not load this model.

Hardware that can hold it

Vendor facts, each read on 2026-11-15:

  • Two 24 GB cards. NVIDIA’s comparison page lists the RTX 3090 and RTX 4090 at 24 GB, drawing 350 W and 450 W per card, and the RTX 5090 at 32 GB and 575 W. Ollama’s FAQ says a model that does not fit on one GPU “will be spread across all the available GPUs”.
  • One 48 GB card. NVIDIA’s professional line has the RTX A6000 and RTX 6000 Ada at 48 GB and 300 W, the RTX PRO 5000 Blackwell at 48 or 72 GB, and the RTX PRO 6000 Blackwell at 96 GB.
  • Unified memory, Apple. The MacBook Pro with M5 Max (40-core GPU) comes with 48, 64 or 128 GB and 614 GB/s of memory bandwidth. The Mac Studio with M5 Ultra goes from 96 to 512 GB, at 1.2 TB/s. macOS limits how much the GPU may hold: see the VRAM guide.
  • Unified memory, NVIDIA. The DGX Spark, the machine we own, has 128 GB and 273 GB/s.

Bandwidth is in that list for a reason. A dense model reads all its weights for every token, so bandwidth divided by file size caps the speed for one user: 273 ÷ 42.52 is 6.4 tokens per second on a Spark, however much memory is free. What to buy at each budget is in our hardware catalogue.

What speed to expect

Published by others. None of these is Llama 3.3 itself: they are the earlier 70B models of the same family at Q4_K_M.

HardwareGenerationSource and conditions
DGX Spark, 128 GB unified4.42 tok/sOllama’s blog, 23 Oct 2025: llama3.1:70b, Ollama 0.12.6, 500 tokens out
Two RTX 309016.29 tok/sXiongjieDai’s benchmark repository: Llama 3 70B, llama-bench, 1,024 tokens, May 2024
Two RTX 409019.06 tok/ssame
RTX A6000, 48 GB14.58 tok/ssame
RTX 6000 Ada, 48 GB18.36 tok/ssame
M3 Max (40-core GPU), 64 GB7.53 tok/ssame
One RTX 3090 or one RTX 4090out of memorysame

The repository’s figures are from 2024 software, the cards on rented machines. Treat them as an order of magnitude.

Measured by us, with the protocol of our 30-minute benchmark guide: one 278-word prompt, 200 tokens out, second run reported. The 8B ran on our first Spark (Ollama 0.30.10), the other three on the second (Ollama 0.34.0, eight slots).

Model (Q4_K_M)FileGenerationFirst tokenReceipt
llama3.1:8b, dense4.92 GB43.7 tok/s0.385 s2026-11-02
mistral-small:24b, dense14.33 GB14.4 tok/s0.36 s2026-11-15
gemma4:26b, mixture of experts17.99 GB68.2 tok/s0.212 s2026-11-02
qwen3.6:35b, mixture of experts23.94 GB75.8 tok/s0.243 ssame file

Mistral Small 24B is the closest thing we have to an anchor: dense, like the 70B, and measured today. If speed fell in proportion to file size, its 14.4 tokens per second would become 4.9 for a 42.52 GB file. That is a sum, not a measurement, and it lands beside Ollama’s published 4.42.

The two fastest rows are the largest files we run. They are mixture-of-experts models, which read only part of their weights per token. A 500-token answer takes about 113 seconds at 4.42 tokens per second, 26 at 19.06 and under 7 at 75.8.

If you run it: the commands and the checks

We did not run these with this model. They are the commands we use daily with others, on Ollama.

  1. Do the sum for your context and slots. Stop here if it does not fit.
  2. Fix the context before the first load. Ollama’s context length page sets the default by memory: 4k under 24 GiB, 32k from 24 to 48, 256k from 48 up.
bash
OLLAMA_CONTEXT_LENGTH=8192 ollama serve
ollama pull llama3.3:70b
ollama run llama3.3:70b "Summarise this clause in three bullet points: ..."
  1. Check where it landed:
bash
ollama ps

The PROCESSOR column should read 100% GPU and CONTEXT should be the value you set. The FAQ’s example of a model that did not fit is 48%/52% CPU/GPU.

  1. Compare the load log with your sum (the command is in the VRAM guide) and time it with your own prompt, discarding the first run.

What goes wrong

  • The default context. On a 48 GB setup a 32k default means 10.74 GB of cache: the sum is 54.26 GB and part of the model goes to the CPU. At the full 131,072 tokens the cache alone is 42.95 GB per slot. Set it yourself.
  • A CPU/GPU split in ollama ps. The CPU layers set the pace. Shorten the context, cut the slots or take a smaller file, in that order. We have no number of our own for a split.
  • Unified memory that runs out. On our machines the symptom was a frozen desktop and NV_ERR_NO_MEMORY in the driver log (see the DGX Spark review). Check available memory first, not total.

When a smaller model is the better choice

More often than the benchmark tables suggest. Three things we measured:

  • Extraction. On 40 synthetic Spanish invoices, gemma4:26b got 360 of 360 fields right; the 7B and 8B models got 321 to 344 (receipt, method). A 17.99 GB file left nothing for a 70B to improve on that test.
  • Spanish writing. Mistral Small 24B handled all three of our office tasks (an email, a summary, an invoice to JSON), which no 7B or 8B model did. Its email took 20.3 seconds against 7.7 for llama3.1:8b (answers, scores). One reader scored them, not a panel.
  • Waiting. The speed table above.

So test an 8B, then a mid-size model, on your own documents. Move to a 70B only if the failure you see is one that size fixes and the wait is acceptable. A 70B earns its place in a nightly queue more easily than in a chat window.

Has it been overtaken?

In releases, yes. What we can source, read on 2026-11-15:

  • Meta itself. Llama 4 arrived on 5 April 2025: Scout (109 billion parameters in total, 17 billion active) and Maverick (400 billion in total). Both are multimodal and larger, so not an easier local option. Meta’s post describes Maverick as high quality at a lower price than Llama 3.3 70B.
  • A smaller Meta model under Apache 2.0. Muse Glimmer 30B appeared in August 2026. Its card names Meta’s Superintelligence Lab as author and gives about 29.6 billion parameters, a January 2026 knowledge cutoff and a 4-bit version under 20 GB. Artificial Analysis calls it Meta’s first open-weights release since Llama 4. The card does not compare it with Llama 3.3 and does not name Spanish among its languages.
  • Other makers. The Gemma 4 page in Ollama’s library reports 82.6% on MMLU Pro for the 26B; Meta’s card gives Llama 3.3 70B 68.9. Different laboratories and conditions: an indication, not a measurement.

We cannot say how Llama 3.3 70B compares with these on your work: we have not run it. We can say that the 26B and 35B models we run have files under 24 GB and are many times faster than the published 70B figure on our class of machine.

Reasons to still choose it: a workflow or fine-tune already validated on it, a need for a text-only model with Spanish on its supported list, or hardware you already own.

Frequent questions

Can I run it on one RTX 4090? Not at a quality worth having. The benchmark repository above records out of memory for Q4_K_M on one 24 GB card, and the only file that fits is rated very low quality.

Is 64 GB of RAM without a GPU enough? The sum is the same, so it should load, but everything runs on the CPU. We have no figure for that and would not plan on it.

Does Q5 or Q8 make it slower? By the bandwidth argument, yes. In Ollama’s Spark table, Q8 files of smaller dense models ran at about two thirds of the Q4 speed. We have no 70B figure.

Limits

We did not run the model: no speed, loaded size or quality judgement of ours for it. The memory sum is verified on models of 7 to 24 billion parameters, not on this one, and the 1 GB of buffers is an allowance. Published speeds are for Llama 3 and 3.1 70B on older software. We have no Mac, dedicated card or split load of our own.

Next steps

Let’s talk for 15 minutes

We do this sum with your real documents and number of users, and test the smaller models first, before recommending any hardware. If you want it for your company, ask for a 15-minute call or see how we work in consulting.

Diagram
Share: LinkedIn X
Veredicto semanal

Get new guides before anyone else

Subscribe and we tell you when new guides, templates and workflows go up. One email a week, no spam.

Already published: 102 guides and 25 templates. All free, no signup.

Bonus: the local-AI starter pack PDF when you subscribe
Once a week No spam Unsubscribe anytime

See what you get