Scaling Data Generation in Vision-and-Language Navigation

TL;DR

This paper introduces a large-scale data generation method for vision-and-language navigation, significantly improving agent performance and generalization across multiple datasets by synthesizing millions of instruction-trajectory pairs from diverse environments.

Contribution

The authors propose a novel data augmentation paradigm that leverages web resources and photorealistic datasets to create extensive training data, enhancing navigation agent performance and generalization.

Findings

Achieved +11% success rate improvement on R2R test split.

Reduced generalization gap to less than 1%.

Set new state-of-the-art results on multiple navigation benchmarks.

Abstract

Recent research in language-guided visual navigation has demonstrated a significant demand for the diversity of traversable environments and the quantity of supervision for training generalizable agents. To tackle the common data scarcity issue in existing vision-and-language navigation datasets, we propose an effective paradigm for generating large-scale data for learning, which applies 1200+ photo-realistic environments from HM3D and Gibson datasets and synthesizes 4.9 million instruction trajectory pairs using fully-accessible resources on the web. Importantly, we investigate the influence of each component in this paradigm on the agent's performance and study how to adequately apply the augmented data to pre-train and fine-tune an agent. Thanks to our large-scale dataset, the performance of an existing agent can be pushed up (+11% absolute with regard to previous SoTA) to a significantly new best of 80% single-run success rate on the R2R test split by simple imitation learning. The long-lasting generalization gap between navigating in seen and unseen environments is also reduced to less than 1% (versus 8% in the previous best method). Moreover, our paradigm also facilitates different models to achieve new state-of-the-art navigation results on CVDN, REVERIE, and R2R in continuous environments.