Interactive Grounded Language Acquisition and Generalization in a 2D World

TL;DR

This paper presents a virtual agent that learns language grounded in visual perception within a 2D maze, capable of generalizing to new words and sentence structures through interactive learning and shared concept detection.

Contribution

The work introduces a novel interactive learning framework that disentangles language grounding from other routines, enabling reliable zero-shot generalization in a 2D environment.

Findings

Outperforms five comparison methods in zero-shot sentence interpretation

Learns from over 1.6 million sentences with diverse vocabulary

Demonstrates human-interpretable intermediate model outputs

Abstract

We build a virtual agent for learning language in a 2D maze-like world. The agent sees images of the surrounding environment, listens to a virtual teacher, and takes actions to receive rewards. It interactively learns the teacher's language from scratch based on two language use cases: sentence-directed navigation and question answering. It learns simultaneously the visual representations of the world, the language, and the action control. By disentangling language grounding from other computational routines and sharing a concept detection function between language grounding and prediction, the agent reliably interpolates and extrapolates to interpret sentences that contain new word combinations or new words missing from training sentences. The new words are transferred from the answers of language prediction. Such a language ability is trained and evaluated on a population of over 1.6 million distinct sentences consisting of 119 object words, 8 color words, 9 spatial-relation words, and 50 grammatical words. The proposed model significantly outperforms five comparison methods for interpreting zero-shot sentences. In addition, we demonstrate human-interpretable intermediate outputs of the model in the appendix.