RUN through the Streets: A New Dataset and Baseline Models for Realistic Urban Navigation

TL;DR

This paper introduces the RUN dataset for urban navigation, combining natural language instructions with real-world dense maps, and evaluates baseline models to understand key architectural factors for success.

Contribution

The paper presents a new dataset for realistic urban navigation and analyzes neural architecture components critical for interpreting navigation instructions.

Findings

Entity abstraction improves accuracy

Attention mechanisms over words and worlds are important

Updating world-state enhances task performance

Abstract

Following navigation instructions in natural language requires a composition of language, action, and knowledge of the environment. Knowledge of the environment may be provided via visual sensors or as a symbolic world representation referred to as a map. Here we introduce the Realistic Urban Navigation (RUN) task, aimed at interpreting navigation instructions based on a real, dense, urban map. Using Amazon Mechanical Turk, we collected a dataset of 2515 instructions aligned with actual routes over three regions of Manhattan. We propose a strong baseline for the task and empirically investigate which aspects of the neural architecture are important for the RUN success. Our results empirically show that entity abstraction, attention over words and worlds, and a constantly updating world-state, significantly contribute to task accuracy.