The Hidden Engineering Challenges of Building LLMs for Non-Latin Languages
The AI industry often describes large language models as multilingual. Open a chatbot, type a question in English, Hindi, Arabic, Japanese, or Thai, and you’ll usually get a reasonable answer. From the outside, it looks like the same system works equally well for everyone.
Under the hood, a different story is unfolding. The same question can cost dramatically different amounts to process depending on the language used. The same context window can hold significantly less information. The same subscription fee can deliver different amounts of value.
This is what I call the “$1 Problem” of multilingual AI: a dollar spent on AI does not buy the same amount of AI in every language.
As companies race to deploy AI globally, understanding this hidden disparity is becoming a real product and cost concern — not just an academic one.
The Myth of Language-Neutral AI
Most commercial AI products charge based on tokens — the units language models use to process text. A token may be a word, part of a word, or a single character, depending on the language and the tokenizer.
For English speakers, tokenization is usually efficient. Common words and phrases are represented compactly because most modern language models were trained primarily on English-heavy datasets. The tokenizer has effectively learned to compress English well.
Most teams assume that if a model works well in English, it will be equally efficient in other languages. Recognizing tokenization challenges can empower AI professionals to address disparities effectively.
Two users can ask the same question and receive the same answer, yet the model may do significantly more work for one of them simply because of the language being used. Since AI providers charge — and operate — based on token usage, the cost of serving those two requests can diverge as well. Most users and product teams never see this happening; the disparity is hidden behind the model’s abstraction layer.
Take a single sentence — “Please review the attached quarterly report and share your feedback by Friday” — and run it through a common tokenizer (e.g., OpenAI’s cl100k_base via tiktoken, or any tokenizer relevant to the model you’re benchmarking) in English, Hindi, Arabic, and Thai. In my experience testing this kind of sentence across scripts, the same content in Hindi or Thai typically comes out to roughly 2–4x the token count of the English original — but you should run this yourself and cite the actual figures, since ratios vary by tokenizer, sentence structure, and script. Toaccuracy improve measurement (pip install tiktoken, consider automating token count tests len(tokens)) multiple languages and scripts, providing concrete data to inform engineering decisions and cost estimates.
Why Languages Consume Different Numbers of Tokens
The root of the problem is tokenization.
Before a language model can understand text, the text must be broken into tokens. Most modern LLMs use variants of Byte Pair Encoding (BPE), SentencePiece, or Unigram tokenization. These approaches work well, but they are heavily shaped by the data used to train them.
When a tokenizer encounters a language that was richly represented during training, it can often encode words efficiently. When it encounters an underrepresented language, the same sentence may fragment into many more pieces — a phenomenon often called token inflation.
The effect isn’t uniform across non-English languages, either. Thai poses a particular challenge because words aren’t separated by spaces, so the tokenizer has to guess at word boundaries. Arabic’s complex morphology means a single root can take dozens of inflected forms, each potentially tokenized differently. Indic scripts like Hindi contain ligatures, combining marks, and character sequences that don’t map cleanly onto the byte patterns a Latin-trained tokenizer has learned to compress.
The result: two people expressing the same thought in different languages can require very different amounts of processing from the model. That difference doesn’t stay confined to research papers — it shows up directly in the cost, speed, and quality of real-world AI products.
The Hidden Cost of Multilingual AI
The first consequence is financial. Every additional token increases inference cost, and at scale, small inefficiencies become meaningful operational expenses. For AI providers, this means higher aggregate serving costs. For enterprises consuming AI APIs, it can mean unexpectedly large bills in regions where users primarily interact in non-Latin languages.
For consumers, the impact is more indirect but no less real. Products that impose token limits — whether on a single request or a context window — effectively give some users less usable capacity than others for the same price.
Imagine two customers paying the same monthly subscription fee. One can fit an entire research document into a model’s context window. The other reaches the limit with substantially less content, because their language requires more tokens to represent the same information. Both purchased the same product. Neither received the same value.
Context Windows Are Not Equal
The recent race toward larger context windows has intensified this issue. Vendors advertise context sizes of 128,000, 200,000, or even millions of tokens — numbers that create the impression every user can process the same amount of information.
In practice, context windows are language-dependent. A 128,000-token window may hold a full legal brief in English while accommodating only a fraction of the equivalent document in Hindi, Arabic, or Thai. For retrieval systems, document analysis, customer support workflows, legal review, and enterprise knowledge management, that gap can materially change what a product can actually do for a given user — not just how much it costs.
The advertised capacity remains the same. The effective capacity does not.
Why Bigger Models Don’t Automatically Solve the Problem
A common assumption in AI is that scale solves everything: more parameters, more GPUs, more training data. Language inequity, however, is not purely a model-size problem — many of the underlying issues originate much earlier in the pipeline.
Training datasets remain heavily skewed toward a small number of languages. English dominates large portions of the public web. High-quality digitized content is abundant for some languages and scarce for others. Evaluation benchmarks disproportionately measure performance in languages that already have strong digital representation, which means teams can ship a model that scores well on the benchmarks that matter to reviewers while still underperforming for the majority of the world’s speakers.
As a result, models inherit the structural biases of the data used to build them. Even highly capable multilingual models often show uneven performance across languages because the underlying data ecosystem is itself uneven.
Beyond Cost: Quality Suffers Too
Tokenization inefficiency is usually framed as a cost problem. It’s also a quality problem.
When information fragments excessively, models must reason across more tokens to reconstruct meaning. Context windows become effectively smaller. Retrieval systems become less precise. Long-form generation becomes harder to sustain coherently.
The consequences surface in places teams don’t always expect to look: search quality, summarization accuracy, translation fidelity, question-answering precision, agent workflows, and retrieval-augmented generation pipelines. The effects can be subtle individually, but they accumulate across the stack — and teams frequently discover them only after launching a product internationally, by which point the architectural choices that created the problem are difficult to reverse.
Building Fairer Multilingual AI
The industry is beginning to recognize these challenges, and the fix starts with measurement. Organizations building global AI products need language-specific visibility, not just aggregate metrics: token usage by language, latency by language, retrieval quality by language, and context-window utilization by language, all measured against test suites that reflect the markets they actually serve rather than the languages that happen to be well-represented in existing benchmarks.
The deeper fix is upstream of any single product decision. Improvements in multilingual AI tend to come less from larger models and more from better corpora for underrepresented languages, better tokenization strategies suited to non-Latin scripts, and evaluation frameworks that treat language parity as a first-class metric rather than an afterthought.
The next decade of AI won’t be defined solely by model scale. It will be defined by how effectively the industry brings high-quality AI experiences to the majority of the world’s languages.
The Next Billion Users
The next billion AI users are unlikely to be English-speaking software engineers in Silicon Valley. They will come from India, Southeast Asia, Africa, the Middle East, and Latin America, interacting with AI in languages that have historically received far less attention from the technology industry.
For these users, multilingual support isn’t a feature. It’s the product.
Companies that understand the economics of language today will be better positioned to serve these markets tomorrow. Those who continue to treat multilingual AI as a translation problem rather than an engineering one may find their models are far less global than they appear.
The future of AI is multilingual. The question is whether the industry’s infrastructure is ready for it.
