NavDreamer: Video Models as Zero-Shot 3D Navigators

TL;DR

NavDreamer introduces a video-based framework leveraging generative models for zero-shot 3D navigation, addressing data scarcity and dynamic environment modeling, with strong generalization demonstrated across diverse tasks.

Contribution

The paper presents NavDreamer, a novel approach using video models as a universal interface for zero-shot 3D navigation, incorporating a sampling-based optimization and inverse dynamics decoding.

Findings

Robust zero-shot generalization across unseen objects and environments

Effective navigation performance on multiple benchmarks

Video models excel in high-level decision-making for navigation

Abstract

Previous Vision-Language-Action models face critical limitations in navigation: scarce, diverse data from labor-intensive collection and static representations that fail to capture temporal dynamics and physical laws. We propose NavDreamer, a video-based framework for 3D navigation that leverages generative video models as a universal interface between language instructions and navigation trajectories. Our main hypothesis is that video's ability to encode spatiotemporal information and physical dynamics, combined with internet-scale availability, enables strong zero-shot generalization in navigation. To mitigate the stochasticity of generative predictions, we introduce a sampling-based optimization method that utilizes a VLM for trajectory scoring and selection. An inverse dynamics model is employed to decode executable waypoints from generated video plans for navigation. To systematically evaluate this paradigm in several video model backbones, we introduce a comprehensive benchmark covering object navigation, precise navigation, spatial grounding, language control, and scene reasoning. Extensive experiments demonstrate robust generalization across novel objects and unseen environments, with ablation studies revealing that navigation's high-level decision-making nature makes it particularly suited for video-based planning.