On the Interplay between Positional Encodings, Morphological Complexity, and Word Order Flexibility

TL;DR

This study investigates how different positional encoding methods affect language model performance across diverse languages, challenging previous assumptions about their interaction with morphological complexity and word order flexibility.

Contribution

It provides a comprehensive analysis of positional encodings across multiple languages, showing no clear interaction with linguistic typology as previously hypothesized.

Findings

No consistent interaction between positional encodings and morphological complexity.

Task, language, and metric choices significantly influence results.

Positional encoding choice has limited impact across diverse languages.

Abstract

Language model architectures are predominantly first created for English and subsequently applied to other languages. It is an open question whether this architectural bias leads to degraded performance for languages that are structurally different from English. We examine one specific architectural choice: positional encodings, through the lens of the trade-off hypothesis: the supposed interplay between morphological complexity and word order flexibility. This hypothesis posits a trade-off between the two: a more morphologically complex language can have a more flexible word order, and vice-versa. Positional encodings are a direct target to investigate the implications of this hypothesis in relation to language modelling. We pretrain monolingual model variants with absolute, relative, and no positional encodings for seven typologically diverse languages and evaluate them on four downstream tasks. Contrary to previous findings, we do not observe a clear interaction between position encodings and morphological complexity or word order flexibility, as measured by various proxies. Our results show that the choice of tasks, languages, and metrics are essential for drawing stable conclusions