Can Language Models Learn Typologically Implausible Languages?

TL;DR

This study investigates whether language models can learn typologically implausible languages, revealing that they show some preferences aligned with natural language patterns, suggesting domain-general biases influence language universals.

Contribution

The paper provides the first large-scale, naturalistic assessment of LMs learning plausible and implausible languages, highlighting their biases and learning dynamics in typologically diverse settings.

Findings

LMs are slower to learn implausible languages.

LMs achieve similar performance on some metrics regardless of plausibility.

Results support the role of domain-general biases in language learning.

Abstract

Grammatical features across human languages show intriguing correlations often attributed to learning biases in humans. However, empirical evidence has been limited to experiments with highly simplified artificial languages, and whether these correlations arise from domain-general or language-specific biases remains a matter of debate. Language models (LMs) provide an opportunity to study artificial language learning at a large scale and with a high degree of naturalism. In this paper, we begin with an in-depth discussion of how LMs allow us to better determine the role of domain-general learning biases in language universals. We then assess learnability differences for LMs resulting from typologically plausible and implausible languages closely following the word-order universals identified by linguistic typologists. We conduct a symmetrical cross-lingual study training and testing LMs on an array of highly naturalistic but counterfactual versions of the English (head-initial) and Japanese (head-final) languages. Compared to similar work, our datasets are more naturalistic and fall closer to the boundary of plausibility. Our experiments show that these LMs are often slower to learn these subtly implausible languages, while ultimately achieving similar performance on some metrics regardless of typological plausibility. These findings lend credence to the conclusion that LMs do show some typologically-aligned learning preferences, and that the typological patterns may result from, at least to some degree, domain-general learning biases.