By the end you will have one HTTP gateway in front of every AI call you make. Each request passes a personal-data detector and a sensitivity header; anything carrying a national ID, an IBAN or an email always goes to a model on your own machine, and only work declared public may leave through a second lane. The local model needs about 9 GB of memory, so a 16 GB laptop with Docker and Ollama should hold it; we measured on a DGX Spark.
What you need
- A computer with 16 GB of RAM. The local lane uses
llama3.1:8b, which took 9.2 GB loaded according toollama pson our machine (Ollama 0.30.10, 2026-11-01). The gateway itself stays under 100 MB. - Docker and Docker Compose. Ours: Docker 29.2.1 and Compose v5.0.2.
- Ollama with
llama3.1:8bpulled (ollama pull llama3.1:8b). - A second OpenAI-compatible endpoint for the public lane. In production that is your external provider. In this guide it is a second local model (
gemma4-32k, 25.8B at Q4_K_M) on our second machine, standing in. We called no paid API, and the numbers for that lane come from a local model, not from a cloud. - Time: about 30 minutes.
Download the full code: hybrid-routing-gateway-2026-11-01.tar.gz (9 KB: gateway, detector, compose file, K3s manifest, test set and benchmark script).
Step 1: The personal-data detector
The detector is a 98-line module in the style of Presidio, but with no model and no network: regular expressions plus the checksum each format carries. It recognises Spanish DNI and NIE (control letter), IBAN (mod 97), email, Spanish phone numbers, Social Security numbers (mod 97) and card numbers (Luhn). The checksum is what stops an eight-digit order number from counting as an ID.
_DNI_LETTERS = "TRWAGMYFPDXBNJZSQVHLCKE"
def _dni_ok(num: str, letter: str) -> bool:
return _DNI_LETTERS[int(num) % 23] == letter.upper()What it does not see, and this needs saying before we go on: names, diagnoses, addresses. “El paciente Juan Pérez tiene diabetes tipo 2” passes clean. That is what the header in step 2 is for, and it is why Presidio layers named-entity recognisers on top of its regexes.
Step 2: The routing logic
The routing is a rule book, not a judgement call. It has three lines, in this order:
def classify(prompt, sensitivity):
entities = sorted({f.entity for f in detect(prompt)})
if sensitivity is not None and sensitivity not in {"public"}:
return "local", f"header sensitivity={sensitivity}", entities
if entities:
return "local", "pii detected: " + ",".join(entities), entities
if sensitivity is None:
return "local", "no sensitivity header (fail closed)", entities
return "bulk", "declared public, no pii found", entitiesHeader first: only public may leave; internal, confidential, pii and regulated stay. Detector second: if it finds anything, the request stays even when the header said public. No header: it stays. Leaving is opt-in and explicit; staying is the default.
Both lanes speak the same format (/v1/chat/completions), so swapping the stand-in for a real provider means changing two environment variables, BULK_URL and BULK_API_KEY. Logs are redacted before they are written: prompt='... con DNI <ES_DNI> confirmando ...'.
Step 3: Start it and check the five routes
docker compose up -d --build
curl -s localhost:8600/healthOutput on our machine:
Container hybrid-gateway-gateway-1 Started
{"status":"ok","local":"http://127.0.0.1:11434/v1","bulk":"http://bulk-host:11434/v1"}Now the five requests that define the behaviour. The first three:
# 1. ID and IBAN in the text, no header
curl -s localhost:8600/v1/route -H 'Content-Type: application/json' \
-d '{"prompt":"Redacta un correo para el cliente con DNI 12345678Z confirmando que su transferencia a ES91 2100 0418 4502 0005 1332 ha llegado.","max_tokens":80}'
# 2. Declared public, no personal data
curl -s localhost:8600/v1/route -H 'Content-Type: application/json' -H 'X-Sensitivity: public' \
-d '{"prompt":"Escribe tres títulos de blog sobre cómo medir tokens por segundo en un modelo local.","max_tokens":80}'
# 3. Declared public, but with an email inside
curl -s localhost:8600/v1/route -H 'Content-Type: application/json' -H 'X-Sensitivity: public' \
-d '{"prompt":"Resume: escríbeme a maria.lopez@example.es cuando esté listo.","max_tokens":40}'Replies, trimmed (2026-11-01):
1: "route":"local", "reason":"pii detected: ES_DNI,IBAN", "model":"llama3.1:8b", "upstream_ms":1160.6
2: "route":"bulk", "reason":"declared public, no pii found", "model":"gemma4-32k:latest", "upstream_ms":871.3
3: "route":"local", "reason":"pii detected: EMAIL", "model":"llama3.1:8b"The fourth, with no header and no personal data (a haiku), returns "reason":"no sensitivity header (fail closed)" and stays local. The fifth, with X-Sensitivity: whatever, returns HTTP 422: the gateway accepts only the five declared values. If request 1 gives you a 502, read docker compose logs gateway: almost always Ollama is not listening on 127.0.0.1:11434.
On n8n: we did not put a webhook in front. The gateway is already an HTTP endpoint and an n8n “HTTP Request” node calls it exactly as curl does; a webhook would add a hop and nothing else.
Step 4: Measure on your machine
The tests/bench.py script in the package sends one warm-up and twenty sequential requests per case at temperature 0, and writes a JSON with p50, p95 and tokens per second:
python3 tests/bench.py http://127.0.0.1:8600 20 bench.json
python3 tests/eval_pii.pyThe second line computes the detector’s precision and recall over tests/pii_testset.jsonl, 56 sentences we labelled by hand (25 with personal data, 31 without). Add your own: every false negative you find is a rule you are missing.
Step 5: The K3s manifest, validated only
The package ships k8s/gateway.yaml: a Deployment, a Service and two NetworkPolicies. The first lets the local-model pod talk to cluster DNS and nothing else, so it has no way out. The second limits the gateway to the two model services and DNS.
kubeconform -strict -summary k8s/gateway.yamlSummary: 4 resources found in 1 file - Valid: 4, Invalid: 0, Errors: 0, Skipped: 0That is all we can claim: the YAML is valid according to kubeconform v0.8.0. We did not deploy it to a cluster, so we will not tell you it works on yours untested.
What we measured
Gateway and local lane on our NVIDIA DGX Spark GB10 (spark-43d5, Ubuntu 24.04, Ollama 0.30.10) with llama3.1:8b Q4_K_M. Public lane: gemma4-32k Q4_K_M on our second DGX Spark GB10 (spark-7180, Ollama 0.34.0) over a direct link. Command: tests/bench.py, 1 warm-up + 20 requests per case, max_tokens 120, 2026-11-02. tok/s = completion tokens / model call time, so it includes prompt processing: a floor, not a peak.
| Case | Lane | Model | p50 (ms) | p95 (ms) | tok/s (median) | Completion tokens |
|---|---|---|---|---|---|---|
| ID, IBAN and phone in the prompt (classify a ticket) | local | llama3.1:8b | 2,219 | 2,249 | 37.4 | 83 |
confidential header (three-sentence summary) | local | llama3.1:8b | 3,133 | 3,163 | 38.3 | 120 |
public header (three titles) | public (local stand-in) | gemma4-32k | 811 | 848 | 66.5 | 54 |
The gateway adds a median of 1.3 to 1.6 ms per request (end-to-end minus model time). The detector over the 56 sentences: precision 0.957 (1 false positive), recall 0.88 (3 false negatives), F1 0.917. Every run, including two from 2026-11-01, is in hybrid-routing-gateway-2026-11-02.json.
What the spec sheet will not tell you
- A shared lane has a p95 you do not control. The same public lane, measured on 2026-11-01 while that machine was serving two other jobs, gave p50 3,861 ms and 14 tok/s, against 811 ms and 66.5 tok/s when idle. An external provider behaves the same way, and you will not see why.
- The local model has rules of its own.
llama3.1:8brefused to write the email in request 1 (“no puedo generar correos electrónicos que contengan información financiera o de identificación personal”). Classifying and summarising the same ticket, it did without complaint. Pick the task, not just the model. - The Docker bridge could not reach the host.
host.docker.internal:11434gaveConnectTimeouton our Ubuntu: the firewall drops that traffic. The compose file usesnetwork_mode: hostand the gateway listens on127.0.0.1:8600only (ss -tlnconfirms it). - Thinking models return empty
content.gemma4on Ollama’s/v1endpoint put 80 tokens intoreasoningand zero into the answer."think": falseis ignored there;"reasoning_effort": "none"works on Ollama 0.34 and the gateway sends it on the public lane. - The detector’s three misses: a phone number with a space after every digit, an ID with the wrong control letter (the checksum rejects it, yet it is still a person’s number) and the name with a diagnosis. The one false positive: a 12-digit meter reading that happens to pass the Social Security mod-97 check.
When not to do this
If everything you process is public, a two-lane gateway is complexity without a prize: one model, local or external, and done. And when the work needs long reasoning over public material, the public lane with a big model is the right call; this design exists so you can use it without personal data travelling along. Before pointing BULK_URL at a real provider you need three more things: a data-processing agreement (GDPR art. 28), the key in an environment variable, and the NetworkPolicy actually applied, not just validated. We measured on a GB10; a 16 GB laptop runs the gateway and llama3.1:8b, but we have not timed it there.
Next steps
- Put numbers on the cost of each lane: cloud vs local AI, your break-even point.
- The same principle with audio: private meeting notes with Whisper on your own hardware.
- If the local lane is tight on memory: pick the right GGUF file for your machine.
Work with us
We size the model and the machine by measuring, not by guessing. If you want to see your own task running on real hardware, book a 15-minute call or see how we work in consulting.