The Roots of Performance Disparity in Multilingual Language Models: Intrinsic Modeling Difficulty or Design Choices?

TL;DR

This survey investigates whether performance gaps in multilingual language models are due to intrinsic linguistic complexity or modeling choices, highlighting how design decisions influence language fairness.

Contribution

It systematically analyzes linguistic features and modeling mechanisms, providing design recommendations to reduce disparities across diverse languages.

Findings

Normalization of segmentation, encoding, and data exposure reduces performance gaps.

Design choices significantly impact multilingual model fairness.

Insights link linguistic features to specific modeling mechanisms.

Abstract

Multilingual language models (LMs) promise broader NLP access, yet current systems deliver uneven performance across the world's languages. This survey examines why these gaps persist and whether they reflect intrinsic linguistic difficulty or modeling artifacts. We organize the literature around two questions: do linguistic disparities arise from representation and allocation choices (e.g., tokenization, encoding, data exposure, parameter sharing) rather than inherent complexity; and which design choices mitigate inequities across typologically diverse languages. We review linguistic features, such as orthography, morphology, lexical diversity, syntax, information density, and typological distance, linking each to concrete modeling mechanisms. Gaps often shrink when segmentation, encoding, and data exposure are normalized, suggesting much apparent difficulty stems from current modeling choices. We synthesize these insights into design recommendations for tokenization, sampling, architectures, and evaluation to support more balanced multilingual LMs.