Learning Vision-and-Language Navigation from YouTube Videos

TL;DR

This paper introduces a novel approach to vision-and-language navigation by leveraging large-scale YouTube house tour videos to create a new dataset, enabling better generalization and state-of-the-art results in navigation tasks.

Contribution

It proposes a method to automatically construct path-instruction pairs from unlabeled videos and pre-trains an agent on this data, addressing limitations of small datasets and improving generalization.

Findings

Achieves state-of-the-art on R2R and REVERIE benchmarks.

Demonstrates effective use of real-world YouTube videos for VLN.

Introduces a trajectory judgment pretext task for layout knowledge mining.

Abstract

Vision-and-language navigation (VLN) requires an embodied agent to navigate in realistic 3D environments using natural language instructions. Existing VLN methods suffer from training on small-scale environments or unreasonable path-instruction datasets, limiting the generalization to unseen environments. There are massive house tour videos on YouTube, providing abundant real navigation experiences and layout information. However, these videos have not been explored for VLN before. In this paper, we propose to learn an agent from these videos by creating a large-scale dataset which comprises reasonable path-instruction pairs from house tour videos and pre-training the agent on it. To achieve this, we have to tackle the challenges of automatically constructing path-instruction pairs and exploiting real layout knowledge from raw and unlabeled videos. To address these, we first leverage an entropy-based method to construct the nodes of a path trajectory. Then, we propose an action-aware generator for generating instructions from unlabeled trajectories. Last, we devise a trajectory judgment pretext task to encourage the agent to mine the layout knowledge. Experimental results show that our method achieves state-of-the-art performance on two popular benchmarks (R2R and REVERIE). Code is available at https://github.com/JeremyLinky/YouTube-VLN