Manipulating language models' training data to study syntactic constraint learning: the case of English passivization

TL;DR

This study investigates how neural language models learn English passivization constraints by manipulating training data, revealing that both frequency and semantics influence passivizability judgments similarly to human learners.

Contribution

It demonstrates that neural models can simulate human-like judgments of passivization restrictions and shows how data manipulation reveals the sources of linguistic knowledge acquisition.

Findings

Neural models' passivization judgments align with human judgments.

Both frequency and semantics independently influence passivizability.

Training data alterations can elucidate sources of linguistic constraints.

Abstract

Grammatical rules in natural languages are often characterized by exceptions. How do language learners learn these exceptions to otherwise general patterns? Here, we study this question through the case study of English passivization. While passivization is in general quite productive, there are cases where it cannot apply (cf. the following sentence is ungrammatical: *One hour was lasted by the meeting). Using neural network language models as theories of language acquisition, we explore the sources of indirect evidence that a learner can leverage to learn whether a verb can be passivized. We first characterize English speakers' judgments of exceptions to the passive, and confirm that speakers find some verbs more passivizable than others. We then show that a neural network language model's verb passivizability judgments are largely similar to those displayed by humans, suggesting that evidence for these exceptions is available in the linguistic input. Finally, we test two hypotheses as to the source of evidence that language models use to learn these restrictions: frequency (entrenchment) and semantics (affectedness). We do so by training models on versions of the corpus that have had sentences of the types implicated by each hypothesis removed, altered, or introduced. We find support for both hypotheses: entrenchment and affectedness make independent contributions to a verb's passivizability. From a methodological point of view, this study highlights the utility of altering a language model's training data for answering questions where complete control over a learner's input is vital.