WorldVLN: Autoregressive World Action Model for Aerial Vision-Language Navigation

TL;DR

WorldVLN introduces an autoregressive world action model for aerial vision-language navigation, enabling agents to predict world states and improve navigation performance in 3D environments.

Contribution

It is the first to formulate aerial VLN as a prediction-driven world-action problem using an autoregressive model and a novel two-stage training framework.

Findings

Outperforms existing baselines with 12%+ success-rate gains.

Enables zero-shot transfer to real drone deployment.

Demonstrates effectiveness on outdoor and indoor benchmarks.

Abstract

Aerial vision-language navigation (VLN) requires agents to follow natural-language instructions through closed-loop perception and action in 3D environments. We argue that aerial VLN can be formulated as a prediction-driven world-action problem: the agent should anticipate latent world evolution and act according to the predicted consequences. To this end, we propose WorldVLN, the first autoregressive world action model for aerial VLN. Unlike full-sequence video-generation world models that generate an entire visual clip, WorldVLN adapts a latent autoregressive video backbone to predict short-horizon world-state transitions and directly decodes them into executable waypoint actions. After each action segment is executed, newly received observations are encoded back into the autoregressive context, enabling closed-loop world-action prediction. We further introduce a two-stage training framework that first grounds the video prior in instruction-conditioned navigation dynamics and then develops Action-aware GRPO, the first reinforcement learning method tailored to autoregressive WAMs, to optimize waypoint decisions through their downstream rollout consequences. On public outdoor and indoor benchmarks, WorldVLN consistently outperforms existing Vision-Language-Action baselines with 12\%+ success-rate gains and larger advantages on challenging cases. It further transfers zero-shot to real drone deployment, suggesting that the proposed WorldVLN offers a promising route for spatial action tasks. Demos and code are available at https://embodiedcity.github.io/WorldVLN/.