Implicit Geometry Representations for Vision-and-Language Navigation from Web Videos

TL;DR

This paper introduces a scalable web video-based framework for vision-and-language navigation that leverages implicit geometry representations from RGB frames, significantly improving performance and robustness in real-world indoor environments.

Contribution

It presents a novel large-scale web video dataset with rich annotations and incorporates implicit geometry representations to enhance spatial understanding without fragile 3D reconstructions.

Findings

Achieves new state-of-the-art results on multiple VLN benchmarks.

Enables robust zero-shot navigation in diverse indoor environments.

Utilizes large-scale web videos effectively for embodied navigation.

Abstract

Vision-and-Language Navigation (VLN) has long been constrained by the limited diversity and scalability of simulator-curated datasets, which fail to capture the complexity of real-world environments. To overcome this limitation, we introduce a large-scale video-instruction framework derived from web-based room tour videos, enabling agents to learn from natural human walking demonstrations in diverse, realistic indoor settings. Unlike existing datasets, our framework integrates both open-ended description-enriched trajectories and action-enriched trajectories reconstructed in 3D, providing richer spatial and semantic supervision. A key extension in this work is the incorporation of implicit geometry representations, which extract spatial cues directly from RGB frames without requiring fragile 3D reconstruction. This approach substantially improves data utilization, alleviates reconstruction failures, and unlocks large portions of previously unusable video data. Comprehensive experiments across multiple VLN benchmarks (CVDN, SOON, R2R, and REVERIE) demonstrate that our method not only sets new state-of-the-art performance but also enables the development of robust zero-shot navigation agents. By bridging large-scale web videos with implicit spatial reasoning, this work advances embodied navigation towards more scalable, generalizable, and real-world applicable solutions.