Anticipating the Unseen Discrepancy for Vision and Language Navigation

TL;DR

This paper introduces DAVIS, a novel semi-supervised framework that improves vision-language navigation by anticipating unseen environment discrepancies through test-time visual consistency and adaptation.

Contribution

It proposes a test-time adaptation method using visual consistency signals and a two-stage learning process to enhance generalization in unseen environments.

Findings

DAVIS outperforms previous state-of-the-art on R2R and RxR benchmarks.

The approach improves decision stability in unseen environments.

Test-time adaptation significantly boosts navigation performance.

Abstract

Vision-Language Navigation requires the agent to follow natural language instructions to reach a specific target. The large discrepancy between seen and unseen environments makes it challenging for the agent to generalize well. Previous studies propose data augmentation methods to mitigate the data bias explicitly or implicitly and provide improvements in generalization. However, they try to memorize augmented trajectories and ignore the distribution shifts under unseen environments at test time. In this paper, we propose an Unseen Discrepancy Anticipating Vision and Language Navigation (DAVIS) that learns to generalize to unseen environments via encouraging test-time visual consistency. Specifically, we devise: 1) a semi-supervised framework DAVIS that leverages visual consistency signals across similar semantic observations. 2) a two-stage learning procedure that encourages adaptation to test-time distribution. The framework enhances the basic mixture of imitation and reinforcement learning with Momentum Contrast to encourage stable decision-making on similar observations under a joint training stage and a test-time adaptation stage. Extensive experiments show that DAVIS achieves model-agnostic improvement over previous state-of-the-art VLN baselines on R2R and RxR benchmarks. Our source code and data are in supplemental materials.