The Structural Sources of Verb Meaning Revisited: Large Language Models Display Syntactic Bootstrapping

TL;DR

This study investigates whether large language models utilize syntactic bootstrapping for verb learning, revealing that their verb representations rely heavily on syntactic cues, especially for mental verbs, paralleling human language acquisition.

Contribution

The paper demonstrates that large language models exhibit syntactic bootstrapping behavior in verb learning, especially for mental verbs, by analyzing their responses to syntactic and co-occurrence perturbations.

Findings

Verb representations degrade more with syntactic cue removal.

Mental verbs are more affected by syntactic perturbations.

Noun representations are more impacted by co-occurrence distortions.

Abstract

Syntactic bootstrapping (Gleitman, 1990) is the hypothesis that children use the syntactic environments in which a verb occurs to learn its meaning. In this paper, we examine whether large language models exhibit a similar behavior. We do this by training RoBERTa and GPT-2 on perturbed datasets where syntactic information is ablated. Our results show that models' verb representation degrades more when syntactic cues are removed than when co-occurrence information is removed. Furthermore, the representation of mental verbs, for which syntactic bootstrapping has been shown to be particularly crucial in human verb learning, is more negatively impacted in such training regimes than physical verbs. In contrast, models' representation of nouns is affected more when co-occurrences are distorted than when syntax is distorted. In addition to reinforcing the important role of syntactic bootstrapping in verb learning, our results demonstrated the viability of testing developmental hypotheses on a larger scale through manipulating the learning environments of large language models.