VISTAv2: World Imagination for Indoor Vision-and-Language Navigation

TL;DR

VISTAv2 introduces a generative world model for indoor vision-and-language navigation that predicts egocentric future views and constructs an online value map for improved planning and navigation accuracy.

Contribution

It presents a novel action-conditioned imagination framework that integrates a vision-language scorer with an online value map, enhancing robustness and interpretability in VLN tasks.

Findings

VISTAv2 outperforms strong baselines on MP3D and RoboTHOR datasets.

Action-conditioned imagination and online value fusion are critical for performance.

The model achieves efficient inference on a single GPU.

Abstract

Vision-and-Language Navigation (VLN) requires agents to follow language instructions while acting in continuous real-world spaces. Prior image imagination based VLN work shows benefits for discrete panoramas but lacks online, action-conditioned predictions and does not produce explicit planning values; moreover, many methods replace the planner with long-horizon objectives that are brittle and slow. To bridge this gap, we propose VISTAv2, a generative world model that rolls out egocentric future views conditioned on past observations, candidate action sequences, and instructions, and projects them into an online value map for planning. Unlike prior approaches, VISTAv2 does not replace the planner. The online value map is fused at score level with the base objective, providing reachability and risk-aware guidance. Concretely, we employ an action-aware Conditional Diffusion Transformer video predictor to synthesize short-horizon futures, align them with the natural language instruction via a vision-language scorer, and fuse multiple rollouts in a differentiable imagination-to-value head to output an imagined egocentric value map. For efficiency, rollouts occur in VAE latent space with a distilled sampler and sparse decoding, enabling inference on a single consumer GPU. Evaluated on MP3D and RoboTHOR, VISTAv2 improves over strong baselines, and ablations show that action-conditioned imagination, instruction-guided value fusion, and the online value-map planner are all critical, suggesting that VISTAv2 offers a practical and interpretable route to robust VLN.