Study finds AI models charge a token premium for many non-English languages

Commercial AI models bill by the token, and the same sentence does not cost the same number of tokens in every language. A study published in September by the Coalition for Language Equity in AI Research (CLEAR) measures that gap. The researchers call it a "tokenizer tax": an extra charge that falls hardest on languages with a small online presence and on scripts that tokenizers rarely saw during training.

The gap matters beyond price. More tokens also mean slower responses and less space in a model's context window, the working memory that holds documents and conversation history. Rest of World reported the findings on 26 October.

What the study measured

A tokenizer splits raw text, images or audio into tokens, the units a model processes and the unit most providers charge for. Tokenizers learn from their training data which character sequences to merge into single tokens. Text that resembles the training data gets compressed efficiently. Text that does not gets broken into many small pieces.

According to CLEAR's write-up, the researchers used FLORES-200, a benchmark of sentences professionally translated into about 200 languages, so each language expresses the same meaning. Even among the 20 most popular languages on the internet, the most expensive language used between 1.5 and 5 times more tokens than the cheapest, depending on the model. Across all 200 languages the spread widened. CLEAR reports a ratio of 2.7 times for the most equitable tokenizer and 17 times in the worst case.

CLEAR gives examples from the GPT-5 tokenizer: a Malayalam sentence needs about twice as many tokens as its English equivalent, and Amharic needs about six times as many. The researchers argue that wordiness does not explain the gap. Shan, the most verbose language in the dataset, is only 42 percent longer than English when counted in characters.

Script matters more than rarity

The study found that how rare a language is does not, on its own, determine its cost. The writing system plays a large role. Asturian has relatively few speakers and no Common Crawl data, yet it does comparatively well. The researchers say it "benefits from the Latin-script advantage, as well as the representation from other related Romance languages like Italian."

Languages written in scripts the tokenizers rarely encountered fared worst, consistently across nearly all models analysed. These include Burmese, Tibetan, Tifinagh, Khmer, Lao and Ol Chiki.

The study puts the economic consequence plainly: "A startup in [Bengaluru] may pay a surcharge on every prompt and completion while its competitor in Austin does not." It adds that higher token counts bring higher latency and reduced effective context capacity, and that "these penalties hit hardest where compute and connectivity are already limited." Its conclusion: "Tokenizer inequity is a constraint on who can participate in the AI economy."

Data, design and disclosure

People building AI products for Indian languages told Rest of World the problem has more than one cause.

Pratyush Choudhury, co-founder of the AI-focused venture fund Activate AI, said non-English languages leave less room for documents and conversation history and need more work to generate an answer. He used a tutoring example: "Retaining the student's earlier mistakes and explanations is part of delivering a good lesson. If that history consumes the context window much faster in one language, the disadvantage reaches into the quality of the service."

Raoul Nanavati, CEO of the Mumbai voice infrastructure startup Navana.ai, pointed to open data efforts. He cited RESPIN, a Gates Foundation funded project with the Indian Institute of Science that collected text and read speech from native speakers of 38 dialects.

Dheemanth Reddy, CEO of Maya Research, said many tokenizers remain optimised for English and other high-resource languages, and that the deeper issue is a lack of high-quality digital data. He warned that data alone will not fix the tokenizer. "Token prices are falling very quickly, so the direct cost impact should become less severe over time," he said, "but actually fixing the tokenization gap will still require labs to deliberately design future tokenizers to handle these languages and scripts better."

Choudhury argued that language efficiency should be a product requirement, alongside accuracy, speed and reliability. "Perfect equality may be unrealistic. Large, persistent disparities should still require an explanation and an improvement plan," he said. He wants every major lab to publish a language-by-language scorecard showing what it costs to complete comparable tasks, and to commit to measurable improvements for its worst-served languages.

What it means for work

For businesses that run AI in Malayalam, Amharic, Burmese or Khmer, the same task can cost several times more than it does in English, and that surcharge recurs on every prompt and every answer. Developers in those markets also get less usable context, so tasks that rely on long documents or long conversation histories, such as tutoring, customer support and document review, run into limits sooner and may degrade in quality. Falling token prices shrink the absolute cost, but they do not close the ratio between languages. Closing it depends on how labs design their tokenizers, and none of the people quoted say any lab has committed to that. Until labs publish per-language cost figures, teams outside English-speaking markets have to measure their own token counts to know what a task will cost.