Grounding Generated Videos in Feasible Plans via World Models

TL;DR

This paper introduces GVP-WM, a planning method that grounds video-generated plans into feasible action sequences using world models, improving physical and temporal consistency in generated videos.

Contribution

The paper proposes a novel approach to ground video plans into feasible actions via world models, enabling physically consistent long-horizon planning from zero-shot video generation.

Findings

Recovers feasible plans from zero-shot generated videos.

Improves temporal and physical consistency in video plans.

Effective in navigation and manipulation tasks.

Abstract

Large-scale video generative models have shown emerging capabilities as zero-shot visual planners, yet video-generated plans often violate temporal consistency and physical constraints, leading to failures when mapped to executable actions. To address this, we propose Grounding Video Plans with World Models (GVP-WM), a planning method that grounds video-generated plans into feasible action sequences using a learned action-conditioned world model. At test-time, GVP-WM first generates a video plan from initial and goal observations, then projects the video guidance onto the manifold of dynamically feasible latent trajectories via video-guided latent collocation. In particular, we formulate grounding as a goal-conditioned latent-space trajectory optimization problem that jointly optimizes latent states and actions under world-model dynamics, while preserving semantic alignment with the video-generated plan. Empirically, GVP-WM recovers feasible long-horizon plans from zero-shot image-to-video-generated and motion-blurred videos that violate physical constraints, across navigation and manipulation simulation tasks.