Learning word-referent mappings and concepts from raw inputs

TL;DR

This paper introduces a neural network model that learns word-referent mappings directly from raw images and speech, demonstrating cross-situational learning, generalization to new instances, and referent localization.

Contribution

It presents the first neural model capable of learning from raw multimodal inputs using self-supervision, addressing key challenges in naturalistic language learning.

Findings

Successfully learns word-referent mappings from ambiguous scenes

Generalizes to novel word instances

Locates referents and exhibits mutual exclusivity preference

Abstract

How do children learn correspondences between the language and the world from noisy, ambiguous, naturalistic input? One hypothesis is via cross-situational learning: tracking words and their possible referents across multiple situations allows learners to disambiguate correct word-referent mappings (Yu & Smith, 2007). However, previous models of cross-situational word learning operate on highly simplified representations, side-stepping two important aspects of the actual learning problem. First, how can word-referent mappings be learned from raw inputs such as images? Second, how can these learned mappings generalize to novel instances of a known word? In this paper, we present a neural network model trained from scratch via self-supervision that takes in raw images and words as inputs, and show that it can learn word-referent mappings from fully ambiguous scenes and utterances through cross-situational learning. In addition, the model generalizes to novel word instances, locates referents of words in a scene, and shows a preference for mutual exclusivity.