Reframing linguistic bootstrapping as joint inference using visually-grounded grammar induction models

TL;DR

This paper demonstrates that joint inference through visually-grounded neural models enhances language acquisition by simultaneously learning syntax and semantics, challenging the view that these are separate, sequential processes.

Contribution

It introduces neural models that unify syntactic and semantic learning, showing joint inference improves grammar induction and lexical understanding.

Findings

Joint learning enhances grammar induction accuracy.

Simultaneous syntax and semantics learning improves interpretation of novel sentences.

Joint inference constrains hypotheses spaces, easing language acquisition.

Abstract

Semantic and syntactic bootstrapping posit that children use their prior knowledge of one linguistic domain, say syntactic relations, to help later acquire another, such as the meanings of new words. Empirical results supporting both theories may tempt us to believe that these are different learning strategies, where one may precede the other. Here, we argue that they are instead both contingent on a more general learning strategy for language acquisition: joint learning. Using a series of neural visually-grounded grammar induction models, we demonstrate that both syntactic and semantic bootstrapping effects are strongest when syntax and semantics are learnt simultaneously. Joint learning results in better grammar induction, realistic lexical category learning, and better interpretations of novel sentence and verb meanings. Joint learning makes language acquisition easier for learners by mutually constraining the hypotheses spaces for both syntax and semantics. Studying the dynamics of joint inference over many input sources and modalities represents an important new direction for language modeling and learning research in both cognitive sciences and AI, as it may help us explain how language can be acquired in more constrained learning settings.