Sim-to-Real Transfer via 3D Feature Fields for Vision-and-Language Navigation

TL;DR

This paper introduces a novel sim-to-real transfer method for vision-and-language navigation, enabling monocular robots to achieve panoramic perception and navigation performance comparable to panoramic models, validated in both simulation and real-world environments.

Contribution

The paper proposes a new approach using 3D feature fields and semantic traversable maps to transfer panoramic VLN capabilities to monocular robots, enhancing real-world navigation.

Findings

Outperforms previous monocular VLN methods in benchmarks

Significantly improves real-world navigation performance

Validated in both simulation and real environments

Abstract

Vision-and-language navigation (VLN) enables the agent to navigate to a remote location in 3D environments following the natural language instruction. In this field, the agent is usually trained and evaluated in the navigation simulators, lacking effective approaches for sim-to-real transfer. The VLN agents with only a monocular camera exhibit extremely limited performance, while the mainstream VLN models trained with panoramic observation, perform better but are difficult to deploy on most monocular robots. For this case, we propose a sim-to-real transfer approach to endow the monocular robots with panoramic traversability perception and panoramic semantic understanding, thus smoothly transferring the high-performance panoramic VLN models to the common monocular robots. In this work, the semantic traversable map is proposed to predict agent-centric navigable waypoints, and the novel view representations of these navigable waypoints are predicted through the 3D feature fields. These methods broaden the limited field of view of the monocular robots and significantly improve navigation performance in the real world. Our VLN system outperforms previous SOTA monocular VLN methods in R2R-CE and RxR-CE benchmarks within the simulation environments and is also validated in real-world environments, providing a practical and high-performance solution for real-world VLN.