Tokenizer: how many tokens your text takes
A model reads neither letters nor words: it reads tokens. See how the tokenizers of the models we run split your text, and how much room the same sentence takes in Spanish and in English. No sign-up and nothing sent: it runs in your browser.
See how a model splits your text
Pick a tokenizer, type or paste some text and see every token with its number.
Nothing downloads until you pick one. The files come from vorluxai.com and your text never leaves your browser.
Pick a tokenizer to start.
The tokens show up here once you pick a tokenizer.
Each colour is one token; hover or tap one to see its number. · is a space, ↵ a line break and ▫ a piece of a character that the next token completes.
The same sentence in Spanish and English
Three office texts and their translation. Your browser counts the tokens of each version with the tokenizer you picked.
| Text | Spanish | English |
|---|---|---|
| Email to a supplier | – | – |
| Privacy notice | – | – |
| Meeting notes | – | – |
| Total | – | – |
Show the texts
- Email to a supplierHola, Marta: te confirmo el pedido de doce filtros de aceite para el jueves. ¿Puedes enviarnos la factura con el IVA desglosado? Gracias y un saludo.Hi Marta, I am confirming the order of twelve oil filters for Thursday. Could you send us the invoice with the VAT shown separately? Thanks and best regards.
- Privacy noticeLos datos de nuestros clientes se guardan en un servidor de la oficina y nunca salen de nuestra red.Our customers' data is stored on a server in the office and never leaves our network.
- Meeting notesEn la reunión del lunes acordamos revisar los contratos pendientes antes de final de mes y responder a todas las reclamaciones abiertas.At Monday's meeting we agreed to review the pending contracts before the end of the month and to answer every open complaint.
Pick a tokenizer and the table fills itself in.
Next step: Split a context budget in the Dev Playground VRAM explorer
What you are looking at
A tokenizer turns text into a list of numbers, one per token. A token can be a whole word, a piece of a word, a space plus what follows it, or a single byte. The model only sees those numbers, and everything you measure in a model is counted in tokens: the context that fits, the memory the cache reserves and the speed of the reply.
Each model family ships its own vocabulary, so the same text gives a different token count in Qwen, Gemma or Llama. That is why an 8k context is not "8,000 words", and why you should count with the tokenizer of the model you will actually run.
The Spanish and English table is not an average of anything: it is three fixed texts, counted in your browser with the tokenizer you pick. If the Spanish column comes out higher for your documents, that margin is context and memory you need to plan for.
Read next: how much memory each context size costs, measured · the VRAM maths, step by step · split a context budget yourself
Runs on tokenizers.js by Hugging Face (Apache-2.0), served from vorluxai.com. Vocabularies: Qwen (Apache-2.0), Gemma 4 (Apache-2.0) and Llama 3 (Llama 3 Community License). Built with Meta Llama 3. Sources, licences and sha256 sums: NOTICE.
Tell us what you want to run
Tell us what you want to run and on what budget. We will tell you which hardware you need, which model fits, and what to expect from it, before you spend anything.
102 free guides · 17 compliance templates