Multilingual Large Language Models do not comprehend all natural languages to equal degrees

TL;DR

This study evaluates the comprehension abilities of multilingual large language models across 12 diverse languages, revealing they perform variably and often better on some low-resource languages than on English, challenging common assumptions.

Contribution

It provides a comprehensive cross-linguistic assessment of LLMs, highlighting their uneven performance and the factors influencing their language comprehension abilities.

Findings

LLMs outperform human baselines in some languages

English is not always the best-performing language for LLMs

Performance varies significantly across language families and resource levels

Abstract

Large Language Models (LLMs) play a critical role in how humans access information. While their core use relies on comprehending written requests, our understanding of this ability is currently limited, because most benchmarks evaluate LLMs in high-resource languages predominantly spoken by Western, Educated, Industrialised, Rich, and Democratic (WEIRD) communities. The default assumption is that English is the best-performing language for LLMs, while smaller, low-resource languages are linked to less reliable outputs, even in multilingual, state-of-the-art models. To track variation in the comprehension abilities of LLMs, we prompt 3 popular models on a language comprehension task across 12 languages, representing the Indo-European, Afro-Asiatic, Turkic, Sino-Tibetan, and Japonic language families. Our results suggest that the models exhibit remarkable linguistic accuracy across typologically diverse languages, yet they fall behind human baselines in all of them, albeit to different degrees. Contrary to what was expected, English is not the best-performing language, as it was systematically outperformed by several Romance languages, even lower-resource ones. We frame the results by discussing the role of several factors that drive LLM performance, such as tokenization, language distance from Spanish and English, size of training data, and data origin in high- vs. low-resource languages and WEIRD vs. non-WEIRD communities.