VISTA: Generative Visual Imagination for Vision-and-Language Navigation

TL;DR

VISTA introduces a generative imagination framework using diffusion models to improve vision-and-language navigation, achieving state-of-the-art results by enabling agents to imagine and align visual goals with observations.

Contribution

The paper proposes VISTA, a novel 'imagine-and-align' approach utilizing diffusion models for visual imagination and structured reasoning in VLN tasks, surpassing previous methods.

Findings

+3.6% Success Rate on R2R benchmark

Sets new state-of-the-art on R2R and RoboTHOR

Highlights importance of imagination and alignment in navigation

Abstract

Vision-and-Language Navigation (VLN) tasks agents with locating specific objects in unseen environments using natural language instructions and visual cues. Many existing VLN approaches typically follow an 'observe-and-reason' schema, that is, agents observe the environment and decide on the next action to take based on the visual observations of their surroundings. They often face challenges in long-horizon scenarios due to limitations in immediate observation and vision-language modality gaps. To overcome this, we present VISTA, a novel framework that employs an 'imagine-and-align' navigation strategy. Specifically, we leverage the generative prior of pre-trained diffusion models for dynamic visual imagination conditioned on both local observations and high-level language instructions. A Perceptual Alignment Filter module then grounds these goal imaginations against current observations, guiding an interpretable and structured reasoning process for action selection. Experiments show that VISTA sets new state-of-the-art results on Room-to-Room (R2R) and RoboTHOR benchmarks, e.g.,+3.6% increase in Success Rate on R2R. Extensive ablation analysis underscores the value of integrating forward-looking imagination, perceptual alignment, and structured reasoning for robust navigation in long-horizon environments.