# Learning To Follow Directions in Street View

## TL;DR

This paper introduces StreetNav, a real-world navigation environment based on Google Street View, where agents learn to follow driving instructions, emphasizing generalization across cities and providing strong baselines and analysis.

## Contribution

It presents a new real-world navigation benchmark using Street View, along with models and analysis to evaluate instruction-following and generalization capabilities.

## Key findings

- Agents can learn to navigate using instructions in real-world environments.
- The dataset enables testing of generalization across multiple cities.
- Baseline models demonstrate promising performance and reveal challenges.

## Abstract

Navigating and understanding the real world remains a key challenge in machine learning and inspires a great variety of research in areas such as language grounding, planning, navigation and computer vision. We propose an instruction-following task that requires all of the above, and which combines the practicality of simulated environments with the challenges of ambiguous, noisy real world data. StreetNav is built on top of Google Street View and provides visually accurate environments representing real places. Agents are given driving instructions which they must learn to interpret in order to successfully navigate in this environment. Since humans equipped with driving instructions can readily navigate in previously unseen cities, we set a high bar and test our trained agents for similar cognitive capabilities. Although deep reinforcement learning (RL) methods are frequently evaluated only on data that closely follow the training distribution, our dataset extends to multiple cities and has a clean train/test separation. This allows for thorough testing of generalisation ability. This paper presents the StreetNav environment and tasks, models that establish strong baselines, and extensive analysis of the task and the trained agents.