Few-shot Object Grounding and Mapping for Natural Language Robot Instruction Following

TL;DR

This paper presents a few-shot learning approach for natural language instruction following in robots, enabling the robot to reason about new objects by using exemplars for grounding and mapping, improving performance on a quadcopter control task.

Contribution

The paper introduces a novel few-shot grounding method and a learned map representation that together allow robots to adapt to new objects during instruction following.

Findings

Outperforms prior state-of-the-art in new object scenarios

Enables reasoning about unseen objects at test-time

Effective in physical quadcopter control tasks

Abstract

We study the problem of learning a robot policy to follow natural language instructions that can be easily extended to reason about new objects. We introduce a few-shot language-conditioned object grounding method trained from augmented reality data that uses exemplars to identify objects and align them to their mentions in instructions. We present a learned map representation that encodes object locations and their instructed use, and construct it from our few-shot grounding output. We integrate this mapping approach into an instruction-following policy, thereby allowing it to reason about previously unseen objects at test-time by simply adding exemplars. We evaluate on the task of learning to map raw observations and instructions to continuous control of a physical quadcopter. Our approach significantly outperforms the prior state of the art in the presence of new objects, even when the prior approach observes all objects during training.