What it is
Every large language model works with a fixed vocabulary of “tokens” rather than with words or letters directly. A tokeniser is the piece of software that converts raw text into this vocabulary before the model sees it, and converts the model’s output back into readable text afterward. Tokens are not a stable unit — depending on the tokeniser’s design, a token might be a whole common word, a fragment of a rarer word, a single character, or a piece of punctuation. OpenAI’s own guidance offers a rough rule of thumb for English: about four characters, or three-quarters of a word, per token (OpenAI Help Center).
Tokenisers are typically built using an algorithm called byte-pair encoding (BPE), which learns its vocabulary from a large training corpus by repeatedly merging the most frequently co-occurring pairs of characters or subwords. Different model families use different tokenisers trained on different data: OpenAI’s GPT models use a tokeniser called tiktoken, Anthropic and other providers use their own variants, and each produces a different token count for identical input text. OpenAI publishes a public tokeniser tool (platform.openai.com/tokenizer) that lets anyone paste in text and see exactly how it gets split.
Because commercial model APIs bill by the token — so many dollars per million input tokens, so many per million output tokens — the tokeniser is not a backstage technical detail. It is the meter the bill runs on. Two tokenisers can disagree substantially on how many tokens a given passage costs, and the same tokeniser can treat two languages very differently.
Why it matters for AI governance and narratives
The reason tokenisation keeps surfacing in coverage of AI pricing announcements is that it decouples the advertised price-per-token from the realised price-per-task. A vendor can cut its per-token rate by a large headline percentage while a change to the underlying tokeniser — one that requires more tokens to encode the same amount of text — erodes some of that saving before the customer ever notices. This is precisely the mechanism behind the gap your observatory’s editorial flagged: a 90% sticker-price cut compressing to a smaller realised saving once a more token-hungry tokeniser is factored in.
This also has a documented linguistic dimension with governance implications. Academic research has found that BPE tokenisers trained predominantly on English text systematically require more tokens to encode other languages for equivalent meaning — one 2023 study is titled, directly, “Language Model Tokenizers Introduce Unfairness Between Languages” (arXiv:2305.15425). Because API costs, context-window budgets, and effective rate limits are all denominated in tokens, this means speakers of some languages pay more, or get less usable context, for the same underlying content — a quiet, infrastructural form of unevenness in how the AI build-out distributes its costs across regions and language communities. A 2025 paper, “Tokenization Multiplicity Leads to Arbitrary Price Variation in LLM-as-a-service” (arXiv:2506.06446), extends the point: because providers are free to choose and change tokenisers, the same nominal per-token price can correspond to meaningfully different real costs across models and over time, with no requirement that providers disclose the comparison.
Key facts and dates
Byte-pair encoding, the dominant tokenisation method for modern LLMs, was adapted for language modelling from older data-compression techniques and became standard practice across GPT, Claude, and most other frontier model families. Major providers maintain separate, proprietary tokenisers that are updated alongside new model releases — which is why a new model version can quietly change token economics even when the headline per-token price falls. OpenAI’s cl100k_base encoding (used for GPT-4/3.5-era models) and its newer o200k_base encoding (used for GPT-4o-era models) are documented examples of this evolution, and researchers have measured how the token cost of non-English text shifted between them.
Measured efficiency gaps are substantial and vary by model generation: one comparison found Chinese text required roughly 24–34% more tokens than English on OpenAI’s newer o200k_base encoding, versus 67% more on the older cl100k_base encoding, while Chinese-optimised models such as Qwen and DeepSeek narrow — but do not eliminate — the gap. Anthropic’s own API pricing is published per million input and output tokens, varying by model tier, with no separate disclosure of tokeniser efficiency relative to prior versions — which is the asymmetry of information the editorial’s sourcing noted.
Where to learn more
- OpenAI: What are tokens and how to count them? — the plainest primary explanation of tokens and a rule-of-thumb conversion to words/characters
- OpenAI Tokenizer tool — lets you tokenise any text yourself and see the count
- Claude Platform Docs: Pricing — Anthropic’s official per-model, per-million-token rate card
- “Language Model Tokenizers Introduce Unfairness Between Languages” — peer-reviewed research documenting the cross-language token-cost gap
- “Tokenization Multiplicity Leads to Arbitrary Price Variation in LLM-as-a-service” — research on how tokeniser choice can obscure true comparative pricing across providers and model versions