In the 2026 τ-Multilingual benchmark, exact account lookup succeeded in 81 to 85 percent of simulated English calls across the GPT and Gemini configurations that the researchers ran repeatedly. The corresponding rates were 26 to 56 percent in Korean and 60 to 74 percent in Mandarin. The researchers found a recurring error in the lookup request. Callers gave the family name first, and agents frequently placed it in the field for the given name. An exact-match query can fail even when the agent has both parts of the name. (τ-Multilingual)
In September 2026, Google announced that Gemini 3.8 Live could automatically switch among 97 supported languages during a conversation. A caller can change languages without changing an application setting. If that caller is trying to verify an account, the agent also has to put the information it heard into the fields the account service accepts. The service receives that structured request, which can fail even if the conversation sounds fluent. (Google’s Gemini 3.8 Live announcement)
What Reaches the Lookup Service
A caller speaking Spanish may give a product name in English because that is how it appears on the box. Switching languages within an utterance is one form of code-switching. For a speech recognizer, the switch can occur in the most consequential part of the request. A product name or order number has to match the value the lookup service recognizes. An otherwise accurate transcript can lose the information the service needs.
The CoSE-E study examined this problem using synthetic speech across five language pairs. The researchers measured how recognition errors affected a downstream system’s ability to answer questions about the utterances. Among cases where code-switching worsened the answers, 57 percent involved corruption of a critical entity, such as an ID or email address. A recognizer can preserve nearly every word and still lose the value that determines the answer. (CoSE-E)
Once the agent recognizes a name, the database still has to match it to a record. An account may store a name in Chinese characters while the agent sends a romanized spelling. Both can refer to the same person, yet an exact-match search treats them as different strings. The result depends on what the model sends and how the application handles differences in script.
In one 30-task Mandarin retail comparison in τ-Multilingual, changing database entries from romanized names to Chinese characters raised task completion from 36.7 to 53.3 percent for the GPT configuration. The Gemini configuration stayed at 53.3 percent under both conditions. The experiment was small, and the two models benefited differently from the same change to the database. (τ-Multilingual, localization tests)
The Language of the Reply
A speaker’s choice of language for a product name can differ from their preference for the rest of the conversation. A Spanish-speaking caller might refer to an English model name and still expect an explanation in Spanish. Inferring that preference requires context from earlier in the call, especially when the latest utterance includes only a name or a part number.
The 2026 Output Language Alignment study examined this distinction in text interactions. It found that models frequently answered in an unexpected language on code-switched prompts and sometimes changed languages within a single response. In speech systems, there can also be a mismatch between the reply text and the audio the caller hears. A written reply can contain the correct amount while the spoken reply renders it incorrectly. (Output Language Alignment)
Task Completion Across Languages
An order correction illustrates how a language switch can affect an entire transaction. A caller supplies a reference number in English and corrects the final digit in Spanish. The order lookup depends on whether the application replaces the earlier number with the corrected one. If the audio was transcribed accurately, an incorrect lookup can still arise when the agent holds on to the first number or puts the correction in the wrong field. A transcription score alone cannot distinguish those failures.
End-to-end benchmarks account for more of that behavior by checking whether the requested operation succeeded. Even there, what counts as completion depends on the setup. In τ-Multilingual, simulated callers received transcripts of the agents’ replies, so a pronunciation error could not alter the caller’s next turn. The authors scored spoken output separately and recorded 1,094 errors involving numbers, dates or currencies across 72,164 scored utterances. A real caller would hear the spoken version. An order can therefore be retrieved successfully while the amount quoted to the caller is wrong. (τ-Multilingual, generation quality and limitations)
The way calls are grouped also affects what the completion rate reveals. The lookup problem in τ-Multilingual was most apparent in the results for Korean and Mandarin. In a service that handles mostly English calls, a comparable failure could affect many speakers of one language while accounting for only a small share of overall traffic.
In summary, The Korean and Mandarin results show where multilingual performance can break down in a routine customer service call. An agent can recognize both parts of a customer’s name, yet the account search can fail because the service receives them in the wrong fields.