LANGUAGE ASYMMETRY IN LLMS: THE IMPACT OF NON-ENGLISH DATA ON ACCURACY AND RELIABILITY

Abstract

This paper investigates language asymmetry in large language models (LLMs) – the tendency for models trained primarily on English data to perform worse on other languages. We conduct a comparative evaluation using a university recommendation task to quantify how an English-centric training corpus impacts accuracy and reliability in non-English contexts. Our experiments confirm a stark disparity: the model’s recommendations for English-language (U.S.) queries were mostly accurate (≈83% completely error-free, with only ~7% containing critical factual errors), whereas recommendations for analogous Asian-context queries were often flawed (only ~46% error-free, with ~21% containing critical errors). Even when augmenting the system with a more powerful OpenAI model, the improvement was pronounced for English data (overall error rate dropping from ~30% to ~12%) but far more limited for Asian data (~60% to ~42%, still leaving ~16% of responses with serious errors). These findings illustrate that current LLMs remain significantly less reliable on non-English inputs, posing challenges for real-world applications (e.g. in education) that require region-specific knowledge.