VAMPO: Policy Optimization for Improving Visual Dynamics in Video Action Models

TL;DR

VAMPO is a post-training framework that enhances visual dynamics in video action models by directly optimizing denoising policies with expert-based rewards, improving manipulation tasks and generalization.

Contribution

It introduces a novel policy optimization method for video models that directly targets visual dynamics, addressing the limitations of likelihood-based training.

Findings

Improves task-relevant visual dynamics in manipulation tasks

Enhances downstream action generation quality

Strengthens model generalization across tasks

Abstract

Video action models are an appealing foundation for Vision--Language--Action systems because they can learn visual dynamics from large-scale video data and transfer this knowledge to downstream robot control. Yet current diffusion-based video predictors are trained with likelihood-surrogate objectives, which encourage globally plausible predictions without explicitly optimizing the precision-critical visual dynamics needed for manipulation. This objective mismatch often leads to subtle errors in object pose, spatial relations, and contact timing that can be amplified by downstream policies. We propose VAMPO, a post-training framework that directly improves visual dynamics in video action models through policy optimization. Our key idea is to formulate multi-step denoising as a sequential decision process and optimize the denoising policy with rewards defined over expert visual dynamics in latent space. To make this optimization practical, we introduce an Euler Hybrid sampler that injects stochasticity only at the first denoising step, enabling tractable low-variance policy-gradient estimation while preserving the coherence of the remaining denoising trajectory. We further combine this design with GRPO and a verifiable non-adversarial reward. Across diverse simulated and real-world manipulation tasks, VAMPO improves task-relevant visual dynamics, leading to better downstream action generation and stronger generalization. The homepage is https://vampo-robot.github.io/VAMPO/.