Entities: regex versus LLM
Pull names, companies, dates, amounts and tax IDs out of a contract with regular expressions, and measure with F1 whether a local model does better.
CONTRACT is a synthetic contract and GOLD the entities a person marked: { text, type } with type in TYPES. Write four functions:
regexNER(text): entities by rules, in order of appearance and without repeats (same text and type): DATE for dd/mm/yyyy and "3 de septiembre de 2026"; MONEY for amounts like 1.250,00 EUR; ID for a CIF (letter, 7 digits, control) and a NIF (8 digits and a letter); ORG for capitalised names ending in S.L., S.L.U. or S.A.; PER for the name after D. or Dña. (two or more capitalised words, without the honorific).
parseEntities(json): from a JSON text {"entities": [...]} return the list of { text, type } with type in upper case, ignoring items without both fields. On invalid JSON, [].
scoreEntities(pred, gold): { tp, fp, fn, precision, recall, f1 } comparing pairs (norm(text), type) as sets; 0 when a denominator is 0.
keySentences(text, n): the n sentences (split on a full stop plus space, except after D., Dña., S.L., S.A. or S.L.U., or on a line break) with the most entities according to regexNER, in their original order: an extractive summary. The preview compares your rules with the model (LIVE), and the last check scores your own Ollama's real reply.
Challenges 0/6
- Rules on one sentence: date, amount, tax ID, company and person
- The rules on the contract: everything but the place, no repeats
- Reads a model's JSON and copes with a broken one
- Precision, recall and F1 as sets
- The two sentences with the most entities, in order
- Scores the real reply of your model (My Ollama) or the sample
function regexNER(text) {
// one regex per type (DATE, MONEY, ID, ORG, PER); PER keeps group 1, the name without D. / Dña.
// collect { text, type, index } with matchAll, sort by index, drop repeats of text + type
return [];
}
function parseEntities(json) {
// JSON.parse inside try; keep items with string text and type; type in upper case; [] on failure
return [];
}
function scoreEntities(pred, gold) {
// sets of norm(text) + "|" + type; tp, fp, fn; precision, recall, f1 (0 when a denominator is 0)
return { tp: 0, fp: 0, fn: 0, precision: 0, recall: 0, f1: 0 };
}
function keySentences(text, n) {
// split on ". " (not after D., Dña., S.L., S.A., S.L.U.) or line breaks, count regexNER(sentence).length, keep the n best in original order
return [];
}
console.log('regex:', scoreEntities(regexNER(CONTRACT), GOLD), 'modelo:', scoreEntities(parseEntities(LIVE), GOLD));The idea
Entity extraction is the best-sold task of "document AI", and the right question is not whether a model can do it, but whether it beats ten regular expressions on your kind of document, and how you measure that. The measure is F1 over (text, type) pairs: precision (of what it marked, how much was right) and recall (of what was there, how much it found).
What we measured, on this contract and on 2026-10-18 on the DGX Spark: the rules you write here reach F1 0.96 (they find everything but "Valencia", which has no fixed shape). qwen3.6:35b reached F1 0.87 (precision 77 %, recall 100 %) in 42.1 s; llama3.1:8b, F1 0.78 in 3.7 s. Three attempts per model, JSON output mode, temperature 0.2.
Look at why the big model loses: it marks "Dña. Marta Ferrer Soler" with the honorific, and the addresses as places. Those are reasonable answers that exact matching punishes. That is a lesson about the metric as much as about the model: define the annotation rule before you score anyone.
Rules win on templated documents; the model wins when the layout changes every time. The honest path is to measure on your documents with your rule, and keep the cheapest option that clears your bar. All of this runs on your machine: neither the contract nor the tax IDs leave it.
Run it on your machine
With Python, on the contract and its entities (CC BY 4.0). The same rules, scored with F1:
curl -O https://vorluxai.com/data/labs/contrato-entidades.json
python3 -c "
import json, re
c = json.load(open('contrato-entidades.json'))
rules = {'DATE': r'\d{2}/\d{2}/\d{4}|\d{1,2} de [a-záéíóú]+ de \d{4}', 'MONEY': r'\d[\d.]*,\d{2} EUR', 'ID': r'\b[A-Z]\d{7}[0-9A-J]\b|\b\d{8}[A-Z]\b',
'ORG': r'[A-ZÁÉÍÓÚ][\wáéíóúñ]*(?: [A-ZÁÉÍÓÚ][\wáéíóúñ]*)* S\.(?:L\.U\.|L\.|A\.)', 'PER': r'(?:D|Dña)\. ((?:[A-ZÁÉÍÓÚÑ][a-záéíóúñ]+)(?: [A-ZÁÉÍÓÚÑ][a-záéíóúñ]+)+)'}
pred = {(m.group(1) if t == 'PER' else m.group(0), t) for t, r in rules.items() for m in re.finditer(r, c['text'])}
gold = {(e['text'], e['type']) for e in c['gold']}
tp = len(pred & gold); p, r = tp / len(pred), tp / len(gold)
print(f'precision {p:.2f} recall {r:.2f} f1 {2*p*r/(p+r):.2f}')
"Go deeper: the Ollama reference →
This in production, with your data? Let's talk for 15 minutes →