By the end of this guide you will have a working RAG in one Python file. Qdrant stores the vectors, bge-m3 turns text into them, and a small local model answers and cites the passages it used. You will also know which settings are worth tuning, because we measured them. It should fit on a 16 GB laptop, though we did not measure one; our numbers come from two DGX Sparks.
What you need
- Docker, for Qdrant.
- Ollama with two models:
ollama pull bge-m3andollama pull qwen2.5:7b-instruct. - Python 3 with
requests. No framework, no API key. - A folder of Markdown or text files. PDFs and Word files need converting to text first; we did not test any converter here.
How the pieces fit
- Chunk: cut each document into short passages.
- Embed: bge-m3 turns each passage into 1,024 numbers.
- Store: Qdrant keeps each vector with its text and its file.
- Retrieve: the question is embedded the same way and Qdrant returns the closest passages.
- Answer: a chat model reads them and cites them by number.
Step 1: start Qdrant
docker run -d --name qdrant -p 127.0.0.1:6333:6333 \
-v qdrant_data:/qdrant/storage qdrant/qdrant:v1.18.2
curl -s localhost:6333/collections
# {"result":{"collections":[]},"status":"ok","time":0.00001912}We pin the version we tested (1.18.2; the newest release on 2026-11-15 was 1.20.1). We publish the port on 127.0.0.1 only, because Qdrant’s quickstart warns that it “starts with no encryption or authentication”. If the port is taken, map another and set QDRANT when you run the script.
Step 2: check what the embedding model returns
curl -s localhost:11434/api/embed -d '{"model": "bge-m3", "truncate": false,
"input": ["Las vacaciones se piden con quince días de antelación.", "¿Cuántos días de vacaciones tengo?"]}' \
| python3 -c "import sys,json; e=json.load(sys.stdin)['embeddings']; print(len(e), len(e[0]))"
# 2 1024Two sentences in, two vectors of 1,024 numbers out. In our run each vector had length 1.0 and the two sentences scored a cosine of 0.69. The fields that matter, from Ollama’s embed reference (read 2026-11-15):
| Field | What it does |
|---|---|
input | One string or a list. A list comes back in the same order. |
truncate | Defaults to true: text beyond the context is cut without an error. Set false to get an error instead. |
keep_alive | How long the model stays in memory after the call. |
options | Runtime settings such as num_ctx. |
bge-m3 needs no prefix on questions: its model card says it “no longer requires adding instructions to the queries”. Other models do.
Step 3: create the collection
The script does this for you. This is the request it sends:
curl -s -X PUT localhost:6333/collections/docs -H 'content-type: application/json' \
-d '{"vectors": {"size": 1024, "distance": "Cosine"}}'
# {"result":true,"status":"ok",...}sizemust equal the model’s dimension: 1,024 for bge-m3.distance:Cosineis what Ollama’s embeddings guide recommends for semantic search. Qdrant also acceptsDot,EuclidandManhattan.- Neither can be changed later. Qdrant’s collections page lists size and distance as fixed once a vector is created. A new embedding model means a new collection.
Step 4: the whole pipeline in one file
Save this as rag_local.py.
import os, sys, uuid, pathlib, requests
OLLAMA = os.environ.get("OLLAMA", "http://localhost:11434")
QDRANT = os.environ.get("QDRANT", "http://localhost:6333")
LLM = os.environ.get("LLM", "qwen2.5:7b-instruct")
KEEP = os.environ.get("KEEP_ALIVE", "5m")
EMBED, DIM = "bge-m3", 1024 # the collection's vector size is the model's
COLL, SIZE, OVERLAP, TOP_K, NUM_CTX = "docs", 1000, 150, 5, 4096
SYSTEM = ("Answer only from the context and cite in square brackets the number of the passage you use. "
"If the answer is not in the context, say: I don't know.")
def chunks(text, size=SIZE, overlap=OVERLAP):
words, out, i = text.split(), [], 0
while i < len(words):
start, cur, n = i, [], 0
while i < len(words) and (not cur or n + len(words[i]) < size):
cur.append(words[i]); n += len(words[i]) + 1; i += 1
out.append(" ".join(cur))
if i >= len(words): break
back = 0
while back < overlap and i > start + 1:
i -= 1; back += len(words[i]) + 1
return out
def embed(texts):
r = requests.post(f"{OLLAMA}/api/embed", json={"model": EMBED, "input": texts, "truncate": False, "keep_alive": KEEP})
r.raise_for_status()
return r.json()["embeddings"]
def qdrant(method, path, body=None):
r = requests.request(method, f"{QDRANT}{path}", json=body)
r.raise_for_status()
return r.json()["result"]
def ingest(folder):
if not qdrant("GET", f"/collections/{COLL}/exists")["exists"]:
qdrant("PUT", f"/collections/{COLL}", {"vectors": {"size": DIM, "distance": "Cosine"}})
qdrant("PUT", f"/collections/{COLL}/index", {"field_name": "doc", "field_schema": "keyword"})
for f in sorted(pathlib.Path(folder).glob("*.md")):
parts = chunks(f.read_text(encoding="utf-8"))
pts = [{"id": str(uuid.uuid5(uuid.NAMESPACE_URL, f"{f.stem}#{i}")), "vector": vec,
"payload": {"doc": f.stem, "n": i, "text": part}}
for i, (vec, part) in enumerate(zip(embed(parts), parts))]
same_doc = {"filter": {"must": [{"key": "doc", "match": {"value": f.stem}}]}}
qdrant("POST", f"/collections/{COLL}/points/delete?wait=true", same_doc) # drop the old version
qdrant("PUT", f"/collections/{COLL}/points?wait=true", {"points": pts})
print(f.stem, len(pts))
def ask(question, k=TOP_K):
hits = qdrant("POST", f"/collections/{COLL}/points/query",
{"query": embed([question])[0], "limit": k, "with_payload": True})["points"]
ctx = "\n\n".join(f"[{i}] ({h['payload']['doc']}) {h['payload']['text']}" for i, h in enumerate(hits, 1))
msgs = [{"role": "system", "content": SYSTEM},
{"role": "user", "content": f"Context:\n{ctx}\n\nQuestion: {question}"}]
r = requests.post(f"{OLLAMA}/api/chat", json={"model": LLM, "messages": msgs, "stream": False,
"keep_alive": KEEP, "options": {"temperature": 0, "num_ctx": NUM_CTX}})
r.raise_for_status()
print(r.json()["message"]["content"], "\n")
for i, h in enumerate(hits, 1):
print(f"[{i}] {h['payload']['doc']}#{h['payload']['n']} score {h['score']:.3f}")
if __name__ == "__main__":
ingest(sys.argv[2]) if sys.argv[1] == "ingest" else ask(sys.argv[2])What each part does, and why:
chunkscuts at word boundaries into pieces of about 1,000 characters that share 150 characters with the previous one. The overlap keeps a sentence that straddles a cut whole in at least one chunk.embedsendstruncate: false. If a chunk is ever too long, you get an error and not a silently shortened vector.- Point ids are derived from the file name and chunk number (Qdrant accepts unsigned integers or UUIDs). Indexing the same file twice overwrites; it does not duplicate.
- The delete-by-filter removes a file’s old chunks before writing the new ones. Without it, a file that got shorter would leave stale chunks behind. The keyword index on
dockeeps that filter fast as the collection grows. askuses Qdrant’s Query API, available since v1.10, and Ollama’s/api/chatwithstream: false.num_ctxis set on every request: five 1,000-character chunks took a median of 1,637 prompt tokens in our measurement below, so 4,096 leaves room. With 2,000-character chunks, raise it to 8,192.
If a file produces hundreds of chunks, send them to /api/embed in batches (our benchmark used 32).
Step 5: index and ask
python3 rag_local.py ingest ./my-docs
# copias-seguridad 1
# gastos 1
# impresora 1
# ...
python3 rag_local.py ask "¿Cuánto me pagan por kilómetro si voy con mi coche?"
# [1] 0,26 euros por kilómetro.
#
# [1] gastos#0 score 0.648
# [2] vacaciones#0 score 0.382
# [3] teletrabajo#0 score 0.371
# ...
python3 rag_local.py ask "¿Cuál es el horario del comedor?"
# No lo sé.
#
# [1] teletrabajo#0 score 0.447
# [2] gastos#0 score 0.438That is real output from 2026-11-15: six short Spanish office documents we wrote for the test, a throwaway Qdrant 1.18.2 container, bge-m3 and llama3.1:8b on Ollama 0.34.0 on our second DGX Spark. We ran the Spanish edition of this script, where only the prompt wording differs. Requests, responses and the six documents are in rag-qdrant-walkthrough-2026-11-15.json.
Nothing in the documents mentions a canteen, the model said so, and Qdrant still returned five passages, the best at 0.447. Vector search always returns its nearest neighbours. Whether they answer the question is for the prompt, and for your evaluation, to settle.
Step 6: keep the index current
- A file changed: run
ingestagain. The second run over our six files left six points, not twelve. - A file was deleted: remove its chunks by filter.
curl -s -X POST 'localhost:6333/collections/docs/points/delete?wait=true' -H 'content-type: application/json' \
-d '{"filter": {"must": [{"key": "doc", "match": {"value": "impresora"}}]}}'
# {"result":{"operation_id":27,"status":"completed"},"status":"ok",...}- You change the embedding model or the chunk size: build a second collection, test it, then repoint an alias. Qdrant applies alias actions atomically, so readers never see a half-built index. Make the script query the alias (
COLL = "docs_live").
curl -s -X POST localhost:6333/collections/aliases -H 'content-type: application/json' -d '{"actions": [
{"delete_alias": {"alias_name": "docs_live"}},
{"create_alias": {"collection_name": "docs_v2", "alias_name": "docs_live"}}]}'
# {"result":true,"status":"ok",...}- Nobody owns it: a RAG goes stale without a sound. Schedule the ingest and name a person.
Step 7: measure it before you trust it
Write questions the way your colleagues would ask them and, for each, note which document or passage answers it. Then score the retriever on its own:
- hit@k: questions whose correct passage is among the first k results.
- MRR: the average of 1 / (rank of the first correct result). Rank 1 scores 1, rank 2 scores 0.5, a miss scores 0.
The full method, with code and the test that tells a gain from noise, is in Evaluate your RAG with 54 questions. To practise, try the top-k retrieval exercise.
What we measured
Run 1, chunk size (2026-11-02). 50 of our Spanish posts (501,917 characters), 60 hand-written questions with their correct document, frozen in rag-embeddings-es-questions-2026-11-02.json. bge-m3 on Ollama 0.34.0 on our second DGX Spark (GB10, 128 GB); Qdrant 1.18.2 and the reranker (BAAI/bge-reranker-v2-m3) on the first. Answers: Qwen2.5-7B-Instruct in BF16 on vLLM, temperature 0, top 5 chunks. The step 4 script uses Ollama’s 4-bit build; these tables were not measured with it. Chunks overlap by 15 %. Raw file: rag-embeddings-es-2026-11-02.json.
| Chunk size | Chunks | hit@1 | hit@5 | MRR | hit@5 + reranker | MRR + reranker |
|---|---|---|---|---|---|---|
| 500 characters | 1,229 | 81.7 % | 96.7 % | 0.885 | 100 % | 0.908 |
| 1,000 characters | 610 | 78.3 % | 96.7 % | 0.862 | 100 % | 0.928 |
| 2,000 characters | 310 | 83.3 % | 100 % | 0.895 | 100 % | 0.947 |
A document counts once, at its best chunk; the reranker re-scores the top 40 chunks. What the 7B model paid (medians over 60 answers, after reranking):
| Chunk size | Prompt tokens | Answer time | Grounding | ”I don’t know” |
|---|---|---|---|---|
| 500 characters | 839 | 3.6 s | 0.87 | 4 of 60 |
| 1,000 characters | 1,637 | 4.0 s | 0.93 | 3 of 60 |
| 2,000 characters | 3,178 | 5.2 s | 0.94 | 2 of 60 |
Grounding is the share of answer sentences that the reranker scores at 0.35 or more against the five chunks. It is a machine check, not a human one.
Run 2, hybrid search and reranking (2026-11-15). A harder set: the Spanish text of the EU AI Act in 761 chunks and 54 hand-written questions, each tied to the exact passage. bge-m3 on Ollama 0.30.10 and the same reranker on vLLM 0.19.0, on our first Spark. Raw file: reranker-es-legal-2026-11-15.json; idle-GPU timings: reranker-es-legal-idle-gpu-2026-11-16.json.
| Method | First place | Top 5 | Top 10 | MRR@10 |
|---|---|---|---|---|
| Embeddings only (bge-m3) | 41 of 54 | 47 | 48 | 0.81 |
| BM25 only (keywords) | 23 of 54 | 31 | 37 | 0.49 |
| Hybrid: both fused with RRF | 34 of 54 | 44 | 48 | 0.72 |
| Embeddings, top 25 reranked | 44 of 54 | 50 | 51 | 0.86 |
| Hybrid, top 50 reranked | 44 of 54 | 50 | 50 | 0.86 |
The decisions that matter
Chunk size and overlap. In run 1 the three sizes differ by one or two questions, and the 95 % margin on a hit@5 of 96.7 % is about ±4.5 points. Retrieval does not pick a winner. The model’s bill does: going from 1,000 to 2,000 characters nearly doubled the prompt (1,637 to 3,178 tokens) and added about 1.3 seconds per answer. On a small box with a short context window, that decides. Overlap stayed at 15 % throughout; we did not vary it.
Top-k. More passages means more chances and a longer prompt. In run 2, embeddings alone had the right passage in the top 5 for 47 questions and in the top 10 for 48. Doubling k bought one question out of 54.
Hybrid search. It adds a keyword search (BM25, over sparse vectors) to the vector search and fuses both rankings. In run 2 it hurt: against embeddings alone, fusion moved the correct passage up in 3 questions and down in 15 (sign test p = 0.0075). BM25 put 14 of 18 literal questions first, but only 2 of 18 written in everyday words. Our BM25 was a plain one, with no stemming and no stop-word list, and legal text is one kind of corpus. Product codes, invoice numbers and surnames are where keywords earn their place: in the walkthrough, the question about error “E-204” scored 0.36 with embeddings and was the only BM25 match.
A reranker. A cross-encoder reads the question and one passage together and returns a relevance score. In run 2 it took first place from 41 to 44 of 54: it won 6 questions and lost 3, and the sign test gives p = 0.51, which is noise at this size. In run 1 it depended on the chunk size. With 1,000 characters the correct document moved up in 12 questions and down in 2 (p = 0.013); with 500 it was 6 up and 5 down. It cost a median of 87 ms for 25 passages on an idle GPU (217 ms on a busy one).
Add hybrid search or a reranker
Both are optional. Each request below ran on 2026-11-15 and is in the walkthrough file.
Hybrid inside Qdrant. Create the collection with a named dense vector and a sparse one:
{"vectors": {"dense": {"size": 1024, "distance": "Cosine"}},
"sparse_vectors": {"bm25": {"modifier": "idf"}}}Store each chunk under both names. Qdrant builds the BM25 vector itself from the text (it is under Inference in its documentation; it worked on our self-hosted 1.18.2):
{"id": 1, "payload": {"doc": "gastos"},
"vector": {"dense": [0.01, 0.02],
"bm25": {"text": "<chunk text>", "model": "qdrant/bm25", "options": {"language": "spanish"}}}}Then query both and fuse them, as in Qdrant’s hybrid queries page:
{"prefetch": [{"query": [0.01, 0.02], "using": "dense", "limit": 20},
{"query": {"text": "<question>", "model": "qdrant/bm25", "options": {"language": "spanish"}},
"using": "bm25", "limit": 20}],
"query": {"rrf": {"k": 60}}, "limit": 5, "with_payload": true}The dense vectors are shortened here; yours have 1,024 values. Set the language: with "spanish" the query “vacación” found the document about “vacaciones”, and with "english" it found nothing. bge-m3 can also produce sparse vectors of its own, but Ollama’s /api/embed returns only the dense ones.
A reranker. Ollama’s API reference lists no rerank endpoint (checked 2026-11-15), so it needs another server. Ours is BAAI/bge-reranker-v2-m3 (Apache 2.0, multilingual) on vLLM, which implements the Jina and Cohere rerank APIs:
curl -s localhost:8201/v1/rerank -H 'content-type: application/json' -d '{"model": "reranker", "top_n": 3,
"query": "¿Cuánto me pagan por kilómetro si voy con mi coche?",
"documents": ["Cada persona de la oficina tiene 23 días laborables de vacaciones al año.",
"El kilometraje con coche propio se paga a 0,26 euros por kilómetro.",
"Los martes son presenciales para todo el mundo."]}'
# "results":[{"index":1,...,"relevance_score":0.992},{"index":2,...,"relevance_score":0.0000164}, ...]Ask Qdrant for 25 passages, rerank them, keep the best five for the prompt. We did not set up the vLLM server as part of this guide; it was already running.
Failures you will hit
Every message below is one we reproduced on 2026-11-15.
Wrong input: Vector dimension error: expected dim: 1024, got 768(Qdrant, HTTP 400). You switched to a 768-dimension model and kept the collection. Build a new one.Not found: Collection `nope` doesn't exist!(404). You asked before indexing, or the name has a typo.Wrong input: Collection `docs` already exists!(409). A second create call. The script checks/existsfirst.model "no-such-model" not found, try pulling it first(Ollama, 404). Runollama pullwith that name.the input length exceeds the context length(Ollama, 400, only withtruncate: false). A chunk is too long. With default options, bge-m3 on our Ollama 0.34.0 took 2,048 tokens and refused 2,049, although the model declares 8,192. Sendingnum_ctxalone changed nothing; with"options": {"num_ctx": 8192, "num_batch": 8192}a 3,002-token input went through. We did not look into why.- No error at all. Without
truncate: false, a 12,000-word input came back as a normal vector built from its first 2,048 tokens.
What the spec sheet won’t tell you
- A hit is not an answer. Even with the right document in the top five, the 7B said “I don’t know” two to four times in 60. The model reads five chunks, not five documents.
- Tables are hard to find. One of the two questions missed most often across the 11 runs of run 1 (6 of 11 left it out of the top five) was “what do I buy for 1,000 euros?”: the answer sits in one row of a hardware table.
- The grounding check has blind spots. One answer quoted its source almost word for word and still scored 0.
- The first stage sets the ceiling. In run 2 the correct passage was among the first 50 candidates for 52 of 54 questions. No reranker can promote a passage it never sees.
- Our sets are easier than your folder. Clean text and questions by the person who read it. Measure yours.
Verdict
- Start at 1,000 to 2,000 characters with 15 % overlap, top 5, vector search only.
- Write 20 to 50 questions before you tune. Without them, every change is a guess.
- Add a reranker or BM25 only if your own questions show a gain. Ours showed a small one for the reranker and a loss for hybrid search on legal text.
When not to do this
If your documents fit in the model’s context (a handful of policies, one manual), skip retrieval and paste them in. If other people will edit the flow, the n8n version is easier to hand over. If the answer needs every row of a spreadsheet added up, RAG is the wrong tool: it retrieves passages, it does not compute over all of them.
FAQ
Do I need a GPU? We measured on GPUs only. bge-m3 held 0.66 GB loaded, so the embedding side is light; the chat model is what sets your memory needs. We have no CPU timings to give you.
Do I need a vector database at all? Not at small scale. Run 2 kept its 761 vectors in a numpy array and each search took a median of 7.9 ms. Qdrant earns its place when you want filters, persistence, updates and several readers.
Can the model make things up anyway? Yes. The prompt asks it to cite and the script prints the sources so you can check. That lowers the risk without removing it.
Does my data leave the machine? Not with this setup: Ollama and Qdrant both listen on localhost. Keep Qdrant’s port off the network unless you add an API key.
Next steps
- Pick the embedding model with the same 60 questions: best local embedding model for Spanish, measured.
- Put the answers in front of your office: Open WebUI for a 10-person office.
- Pull fields out of your documents as JSON before you index them: Ollama structured output, 40 invoices measured.
- Let an agent call your RAG as a tool: n8n and MCP, or start with three local agents for an SME.
Work with us
We size the model and the machine by measuring, not by guessing. If you want to see your own documents answered on real hardware, book a free 15-minute call or see how we work in consulting.