Lookahead Exploration with Neural Radiance Representation for Continuous Vision-Language Navigation

TL;DR

This paper introduces a novel hierarchical neural radiance model for vision-language navigation that improves future environment prediction, enabling more effective lookahead exploration and path planning in 3D environments.

Contribution

It proposes a pre-trained hierarchical neural radiance model to produce robust semantic features for future environments, enhancing lookahead navigation planning.

Findings

Outperforms existing methods on VLN-CE datasets

Efficient parallel evaluation of future paths

Robust semantic feature representation for environments

Abstract

Vision-and-language navigation (VLN) enables the agent to navigate to a remote location following the natural language instruction in 3D environments. At each navigation step, the agent selects from possible candidate locations and then makes the move. For better navigation planning, the lookahead exploration strategy aims to effectively evaluate the agent's next action by accurately anticipating the future environment of candidate locations. To this end, some existing works predict RGB images for future environments, while this strategy suffers from image distortion and high computational cost. To address these issues, we propose the pre-trained hierarchical neural radiance representation model (HNR) to produce multi-level semantic features for future environments, which are more robust and efficient than pixel-wise RGB reconstruction. Furthermore, with the predicted future environmental representations, our lookahead VLN model is able to construct the navigable future path tree and select the optimal path via efficient parallel evaluation. Extensive experiments on the VLN-CE datasets confirm the effectiveness of our method.