Vocabulary shapes cross-lingual variation of word-order learnability in language models

TL;DR

This study investigates how vocabulary structure influences the learnability of word order in language models across different languages, revealing vocabulary as a key factor in cross-lingual variation.

Contribution

It demonstrates that vocabulary structure, rather than language typology, predicts word-order learnability in transformer models across languages.

Findings

Vocabulary structure predicts surprisal across languages.

Irregular word order increases model surprisal.

Vocabulary features outweigh language typology in learnability.

Abstract

Why do some languages like Czech permit free word order, while others like English do not? We address this question by pretraining transformer language models on a spectrum of synthetic word-order variants of natural languages. We observe that greater word-order irregularity consistently raises model surprisal, indicating reduced learnability. Sentence reversal, however, affects learnability only weakly. A coarse distinction of free- (e.g., Czech and Finnish) and fixed-word-order languages (e.g., English and French) does not explain cross-lingual variation. Instead, the structure of the word and subword vocabulary strongly predicts the model surprisal. Overall, vocabulary structure emerges as a key driver of computational word-order learnability across languages.