Grounding Complex Navigational Instructions Using Scene Graphs

TL;DR

This paper introduces a new dataset of complex navigation instructions paired with scene graphs, enabling training of reinforcement learning agents to understand and execute natural language commands in visual environments.

Contribution

It adapts the CLEVR dataset to generate complex instructions and scene graphs, and demonstrates training an agent in VizDoom to follow these instructions.

Findings

Successfully generated a supervised dataset for navigation tasks

Trained an agent to interpret complex language instructions in VizDoom

Showed the feasibility of scene graph-based instruction grounding

Abstract

Training a reinforcement learning agent to carry out natural language instructions is limited by the available supervision, i.e. knowing when the instruction has been carried out. We adapt the CLEVR visual question answering dataset to generate complex natural language navigation instructions and accompanying scene graphs, yielding an environment-agnostic supervised dataset. To demonstrate the use of this data set, we map the scenes to the VizDoom environment and use the architecture in \citet{gatedattention} to train an agent to carry out these more complex language instructions.