# What Kind of Language Is Hard to Language-Model?

## TL;DR

This study investigates the difficulty of modeling various languages with neural networks, introducing a new statistical model and analyzing 69 languages to understand what makes some languages harder to model.

## Contribution

It extends prior work by applying a novel mixed-effects model to a larger language set, revealing new insights into factors influencing language modeling difficulty.

## Key findings

- Translationese is not easier to model than native language.
- Morphological complexity is not the main factor in language difficulty.
- Simpler data statistics may better explain modeling complexity.

## Abstract

How language-agnostic are current state-of-the-art NLP tools? Are there some types of language that are easier to model with current methods? In prior work (Cotterell et al., 2018) we attempted to address this question for language modeling, and observed that recurrent neural network language models do not perform equally well over all the high-resource European languages found in the Europarl corpus. We speculated that inflectional morphology may be the primary culprit for the discrepancy. In this paper, we extend these earlier experiments to cover 69 languages from 13 language families using a multilingual Bible corpus. Methodologically, we introduce a new paired-sample multiplicative mixed-effects model to obtain language difficulty coefficients from at-least-pairwise parallel corpora. In other words, the model is aware of inter-sentence variation and can handle missing data. Exploiting this model, we show that "translationese" is not any easier to model than natively written language in a fair comparison. Trying to answer the question of what features difficult languages have in common, we try and fail to reproduce our earlier (Cotterell et al., 2018) observation about morphological complexity and instead reveal far simpler statistics of the data that seem to drive complexity in a much larger sample.