Third-party files served by vorluxai.com for the tokenizer tool (/tools/tokenizer/) ================================================================================== tokenizers.min.mjs tokenizers.js 0.2.0 by Hugging Face, https://github.com/huggingface/tokenizers.js Licensed under the Apache License, Version 2.0 (LICENSE-tokenizers.js.txt). Unmodified. qwen2.5-qwen3/tokenizer.json Tokenizer of Qwen3 (identical vocabulary and merges to Qwen2.5), Copyright 2024 Alibaba Cloud, from https://huggingface.co/Qwen/Qwen3-8B. Licensed under the Apache License, Version 2.0 (qwen2.5-qwen3/LICENSE-qwen.txt). Modified: JSON whitespace removed; content unchanged. gemma4/tokenizer.json Tokenizer of Gemma 4 by Google DeepMind, from https://huggingface.co/google/gemma-4-26B-A4B-it. Licensed under the Apache License, Version 2.0 (gemma4/LICENSE-gemma4-apache-2.0.txt, https://ai.google.dev/gemma/docs/gemma_4_license). Modified: JSON whitespace removed; content unchanged. llama3/tokenizer.json Meta Llama 3 is licensed under the Meta Llama 3 Community License, Copyright © Meta Platforms, Inc. All Rights Reserved. Built with Meta Llama 3. Converted to the Hugging Face tokenizer.json format from Meta's tokenizer.model at https://github.com/meta-llama/llama-models/tree/main/models/llama3 . Licence: llama3/LICENSE-llama3.txt; Acceptable Use Policy: llama3/USE_POLICY-llama3.md. Sources, sizes and sha256 sums: README.md in this folder.