What it is
Before a large language model can read or write anything, it has to convert text into numbers. It does this by breaking the text into chunks called tokens — not quite words, not quite characters, but statistically common fragments learned from the model’s training data. A frequent English word like “the” or “hello” typically becomes a single token. A rarer word, a technical term, or text in a script the tokenizer saw less often during training gets split into several tokens. Anthropic’s own documentation describes tokens as “the smallest units the model reads and generates,” following statistical patterns rather than linguistic ones (Anthropic, Claude Platform Docs).
The algorithms that do this splitting — Byte Pair Encoding, WordPiece, and similar schemes — build their vocabulary by finding the most common recurring sequences in a training corpus and giving each one its own token. Because that training corpus is overwhelmingly English-language text scraped from the web, English words compress efficiently into few tokens, while text in other scripts and languages fragments into more of them. A 2023 study by Aleksandar Petrov and colleagues at Oxford, presented at NeurIPS, measured this directly: the same sentence translated across languages can require up to 15 times more tokens depending on the language, and the disparity persists even in tokenizers explicitly built for multilingual use (Petrov et al., “Language Model Tokenizers Introduce Unfairness Between Languages,” NeurIPS 2023).
Because commercial AI providers, including Anthropic, charge by the token for both input and output, this fragmentation translates directly into a price difference. A prompt in Russian, Arabic, Hindi, or Tamil can cost multiple times more than the identical prompt in English — not because the model does more work understanding the meaning, but because its tokenizer was shaped by an English-dominant training diet.
Why it matters for AI governance and narratives
Tokenisation sits at an unusual intersection of technical infrastructure and political economy. It is not a policy choice made by a regulator or a pricing decision made by a sales team — it is an emergent property of how a model’s vocabulary was built — yet it produces an outcome that looks exactly like a policy choice: users in some language communities pay more for the same service. This makes it a useful case for observing how technical artifacts get reframed as narratives. A outlet can describe the same fact — tokenizer inefficiency on non-English scripts — as evidence of indifference, as a byproduct of engineering history, or as a deliberate tax, depending on which framing serves its argument.
It also illustrates a recurring pattern in AI governance debates: a measurable technical disparity (token count per equivalent sentence) becomes a proxy for a much larger claim (linguistic inequality, digital sovereignty, platform bias) that the underlying data can support but not fully prove on its own. Tracking how single data points about tokenisation get cited — and by whom, and toward what argument — is one way to watch the narrative contest over whether AI systems are neutral infrastructure or instruments that encode existing global asymmetries.
Key facts and dates
The unfairness was documented formally in Petrov et al.’s NeurIPS 2023 paper, which found tokenization-length disparities of up to 15x across languages and noted consequences beyond cost: processing time, latency, and the amount of content that fits in a model’s context window all scale with token count, so inefficient tokenization also means a non-English user gets a shorter effective context window for the same budget. Independent reproductions since then — including open benchmarks comparing tokenizers across roughly a dozen languages — consistently find Cyrillic, Arabic, and South Asian scripts costing 1.5 to 3 times more tokens than English for equivalent text, with some comparisons of individual model tokenizers reporting even higher multiples; results vary by methodology, text domain, and which tokenizer version is tested, so exact multipliers should be treated as approximate rather than fixed. Anthropic’s Claude Opus 4.7 introduced a new tokenizer that itself increased token counts by roughly 30% across the board relative to earlier Claude models, a reminder that these ratios shift with each tokenizer redesign rather than being a fixed property of a language.
No tokenizer in wide commercial use today is free of this effect; the disparity is a function of training-corpus composition, not of any single company’s policy, though companies retain full discretion over per-token pricing regardless of tokenizer efficiency.
Where to learn more
- Language Model Tokenizers Introduce Unfairness Between Languages (Petrov et al., NeurIPS 2023, arXiv) — the primary academic measurement of cross-language tokenization disparity.
- Anthropic: Token Counting documentation — official explanation of how Claude’s tokenizer works and how to count tokens before sending a request.
- The Multilingual Token Tax (getnadir.com) — accessible overview connecting tokenizer mechanics to real-world API cost differences across languages.
- llm-language-token-tax (GitHub, reproducible benchmark) — open-source tiktoken-based benchmark showing 1.5x–3.3x token cost ratios across eight languages, for readers who want to verify the effect themselves.