One Agent to Guide Them All: Empowering MLLMs for Vision-and-Language Navigation via Explicit World Representation

TL;DR

This paper introduces a decoupled framework for vision-and-language navigation that uses an interactive metric world representation with MLLMs, achieving state-of-the-art zero-shot performance and effective sim-to-real transfer across different robotic platforms.

Contribution

It proposes a novel decoupled design with an interactive metric world representation and counterfactual reasoning, enhancing MLLMs' decision-making in navigation tasks.

Findings

Achieved 48.8% SR in R2R-CE benchmark.

Achieved 42.2% SR in RxR-CE benchmark.

Demonstrated successful zero-shot sim-to-real transfer on diverse robots.

Abstract

A navigable agent needs to understand both high-level semantic instructions and precise spatial perceptions. Building navigation agents centered on Multimodal Large Language Models (MLLMs) demonstrates a promising solution due to their powerful generalization ability. However, the current tightly coupled design dramatically limits system performance. In this work, we propose a decoupled design that separates low-level spatial state estimation from high-level semantic planning. Unlike previous methods that rely on predefined, oversimplified textual maps, we introduce an interactive metric world representation that maintains rich and consistent information, allowing MLLMs to interact with and reason on it for decision-making. Furthermore, counterfactual reasoning is introduced to further elicit MLLMs' capacity, while the metric world representation ensures the physical validity of the produced actions. We conduct comprehensive experiments in both simulated and real-world environments. Our method establishes a new zero-shot state-of-the-art, achieving 48.8\% Success Rate (SR) in R2R-CE and 42.2\% in RxR-CE benchmarks. Furthermore, to validate the versatility of our metric representation, we demonstrate zero-shot sim-to-real transfer across diverse embodiments, including a wheeled TurtleBot 4 and a custom-built aerial drone. These real-world deployments verify that our decoupled framework serves as a robust, domain-invariant interface for embodied Vision-and-Language navigation.