Benchmarking Concept-Spilling Across Languages in LLMs

TL;DR

This paper introduces a new benchmark and evaluation framework to measure how well multilingual LLMs handle polysemous words across languages, revealing variation in semantic robustness and bias.

Contribution

It presents a novel comparative framework and benchmark for evaluating multilingual semantic robustness and bias in LLMs across multiple languages.

Findings

Significant variation in semantic robustness across models and languages.

Stronger models produce more true meanings before failing.

Benchmark enables principled ranking of models without causal error attribution.

Abstract

Multilingual Large Language Models (LLMs) exhibit remarkable cross-lingual abilities, yet often exhibit a systematic bias toward the representations from other languages, resulting in semantic interference when generating content in non-English languages$-$a phenomenon we define as language spilling. This paper presents a novel comparative framework for evaluating multilingual semantic robustness by systematically measuring how models handle polysemous words across languages. Our methodology provides a relative measure of model performance: when required to generate exactly five meanings, both strong and weak models may resort to meanings from dominant languages, but semantically stronger models do so later in the generation sequence, producing more true meanings from the target language before failing, while weaker models resort to dominant-language meanings earlier in the sequence. We evaluate a diverse set of open and closed multilingual LLMs using a structured meaning generation task across nine languages, employing a carefully curated benchmark of 100 high-polysemy English words. Our findings reveal significant variation in semantic robustness across both models and languages, providing a principled ranking system for model comparison without requiring definitive causal attribution of error sources. We contribute both a scalable comparative benchmark for multilingual semantic evaluation and a rigorous validation pipeline$-$critical tools for developing more linguistically balanced AI systems.