DREAMWALKER: Mental Planning for Continuous Vision-Language Navigation

TL;DR

DREAMWALKER introduces a world model-based approach for vision-language navigation, enabling agents to perform mental planning and simulate future scenarios internally, leading to more strategic and interpretable navigation behaviors.

Contribution

The paper presents a novel world model for VLN-CE that allows for internal simulation of future actions, improving planning and decision transparency.

Findings

Outperforms existing model-free VLN-CE agents in navigation tasks.

Enables strategic planning through mental experiments.

Improves decision transparency and interpretability.

Abstract

VLN-CE is a recently released embodied task, where AI agents need to navigate a freely traversable environment to reach a distant target location, given language instructions. It poses great challenges due to the huge space of possible strategies. Driven by the belief that the ability to anticipate the consequences of future actions is crucial for the emergence of intelligent and interpretable planning behavior, we propose DREAMWALKER -- a world model based VLN-CE agent. The world model is built to summarize the visual, topological, and dynamic properties of the complicated continuous environment into a discrete, structured, and compact representation. DREAMWALKER can simulate and evaluate possible plans entirely in such internal abstract world, before executing costly actions. As opposed to existing model-free VLN-CE agents simply making greedy decisions in the real world, which easily results in shortsighted behaviors, DREAMWALKER is able to make strategic planning through large amounts of ``mental experiments.'' Moreover, the imagined future scenarios reflect our agent's intention, making its decision-making process more transparent. Extensive experiments and ablation studies on VLN-CE dataset confirm the effectiveness of the proposed approach and outline fruitful directions for future work.