Bridging the Gap Between Learning in Discrete and Continuous Environments for Vision-and-Language Navigation

TL;DR

This paper introduces a waypoint predictor to bridge the gap between discrete and continuous environments in vision-and-language navigation, significantly improving agent performance and generalization across different navigation setups.

Contribution

The paper proposes a novel waypoint predictor trained with refined environment graphs to enable high-level action agents to operate effectively in continuous environments.

Findings

Agents with predicted waypoints outperform low-level action agents in continuous environments.

The method reduces the discrete-to-continuous gap by over 11% in SPL.

State-of-the-art results achieved on R2R-CE and RxR-CE datasets.

Abstract

Most existing works in vision-and-language navigation (VLN) focus on either discrete or continuous environments, training agents that cannot generalize across the two. The fundamental difference between the two setups is that discrete navigation assumes prior knowledge of the connectivity graph of the environment, so that the agent can effectively transfer the problem of navigation with low-level controls to jumping from node to node with high-level actions by grounding to an image of a navigable direction. To bridge the discrete-to-continuous gap, we propose a predictor to generate a set of candidate waypoints during navigation, so that agents designed with high-level actions can be transferred to and trained in continuous environments. We refine the connectivity graph of Matterport3D to fit the continuous Habitat-Matterport3D, and train the waypoints predictor with the refined graphs to produce accessible waypoints at each time step. Moreover, we demonstrate that the predicted waypoints can be augmented during training to diversify the views and paths, and therefore enhance agent's generalization ability. Through extensive experiments we show that agents navigating in continuous environments with predicted waypoints perform significantly better than agents using low-level actions, which reduces the absolute discrete-to-continuous gap by 11.76% Success Weighted by Path Length (SPL) for the Cross-Modal Matching Agent and 18.24% SPL for the Recurrent VLN-BERT. Our agents, trained with a simple imitation learning objective, outperform previous methods by a large margin, achieving new state-of-the-art results on the testing environments of the R2R-CE and the RxR-CE datasets.