A Deep Compositional Framework for Human-like Language Acquisition in Virtual Environment

TL;DR

This paper presents a deep, compositional framework enabling a virtual agent to learn language and navigation in a 2D maze, achieving zero-shot command execution through grounded, modular learning.

Contribution

It introduces an end-to-end deep learning approach that learns visual, linguistic, and action representations simultaneously, enabling zero-shot language understanding in a virtual environment.

Findings

Agent can execute zero-shot commands involving new word combinations

Agent understands new object concepts learned from other tasks

Framework visualizes intermediate representations showing comprehension

Abstract

We tackle a task where an agent learns to navigate in a 2D maze-like environment called XWORLD. In each session, the agent perceives a sequence of raw-pixel frames, a natural language command issued by a teacher, and a set of rewards. The agent learns the teacher's language from scratch in a grounded and compositional manner, such that after training it is able to correctly execute zero-shot commands: 1) the combination of words in the command never appeared before, and/or 2) the command contains new object concepts that are learned from another task but never learned from navigation. Our deep framework for the agent is trained end to end: it learns simultaneously the visual representations of the environment, the syntax and semantics of the language, and the action module that outputs actions. The zero-shot learning capability of our framework results from its compositionality and modularity with parameter tying. We visualize the intermediate outputs of the framework, demonstrating that the agent truly understands how to solve the problem. We believe that our results provide some preliminary insights on how to train an agent with similar abilities in a 3D environment.