Do Language Models Learn Position-Role Mappings?

TL;DR

This study investigates whether pretrained language models understand position-role mappings in syntax, demonstrating they recognize role distinctions and generalize this knowledge across different structures, with some limitations on novel verbs.

Contribution

It provides evidence that models like BERT and RoBERTa learn shared position-role mappings and can generalize this knowledge across paradigms, revealing insights into their syntactic understanding.

Findings

Models recognize theme and recipient roles in ditransitive constructions.

Fine-tuning enables models to transfer role knowledge across paradigms.

Performance drops with novel ditransitive verbs, indicating lexical specificity.

Abstract

How is knowledge of position-role mappings in natural language learned? We explore this question in a computational setting, testing whether a variety of well-performing pertained language models (BERT, RoBERTa, and DistilBERT) exhibit knowledge of these mappings, and whether this knowledge persists across alternations in syntactic, structural, and lexical alternations. In Experiment 1, we show that these neural models do indeed recognize distinctions between theme and recipient roles in ditransitive constructions, and that these distinct patterns are shared across construction type. We strengthen this finding in Experiment 2 by showing that fine-tuning these language models on novel theme- and recipient-like tokens in one paradigm allows the models to make correct predictions about their placement in other paradigms, suggesting that the knowledge of these mappings is shared rather than independently learned. We do, however, observe some limitations of this generalization when tasks involve constructions with novel ditransitive verbs, hinting at a degree of lexical specificity which underlies model performance.