Are All Languages Equally Hard to Language-Model?

TL;DR

This paper introduces an evaluation framework for fair comparison of language models across 21 languages, revealing that linguistic complexity like inflectional morphology affects model performance.

Contribution

It develops a cross-linguistic evaluation method using translated text and demonstrates the impact of morphological complexity on language modeling accuracy.

Findings

Performance varies across languages due to morphological complexity

Inflectional morphology significantly affects prediction difficulty

Both n-gram and LSTM models show similar patterns of difficulty

Abstract

For general modeling methods applied to diverse languages, a natural question is: how well should we expect our models to work on languages with differing typological profiles? In this work, we develop an evaluation framework for fair cross-linguistic comparison of language models, using translated text so that all models are asked to predict approximately the same information. We then conduct a study on 21 languages, demonstrating that in some languages, the textual expression of the information is harder to predict with both $n$-gram and LSTM language models. We show complex inflectional morphology to be a cause of performance differences among languages.