mimic-video: Video-Action Models for Generalizable Robot Control Beyond VLAs

TL;DR

mimic-video introduces a video-based action model that enhances robot control by capturing physical dynamics during pretraining, leading to significant improvements in efficiency and performance over traditional vision-language models.

Contribution

the paper proposes mimic-video, a novel video-action model that integrates a pretrained video model with an inverse dynamics decoder, enabling better physical understanding for robotic manipulation.

Findings

achieves state-of-the-art results in robotic tasks

improves sample efficiency by 10x

reduces convergence time by 2x

Abstract

Prevailing Vision-Language-Action Models (VLAs) for robotic manipulation are built upon vision-language backbones pretrained on large-scale, but disconnected static web data. As a result, despite improved semantic generalization, the policy must implicitly infer complex physical dynamics and temporal dependencies solely from robot trajectories. This reliance creates an unsustainable data burden, necessitating continuous, large-scale expert data collection to compensate for the lack of innate physical understanding. We contend that while vision-language pretraining effectively captures semantic priors, it remains blind to physical causality. A more effective paradigm leverages video to jointly capture semantics and visual dynamics during pretraining, thereby isolating the remaining task of low-level control. To this end, we introduce mimic-video, a novel Video-Action Model (VAM) that pairs a pretrained Internet-scale video model with a flow matching-based action decoder conditioned on its latent representations. The decoder serves as an Inverse Dynamics Model (IDM), generating low-level robot actions from the latent representation of video-space action plans. Our extensive evaluation shows that our approach achieves state-of-the-art performance on simulated and real-world robotic manipulation tasks, improving sample efficiency by 10x and convergence speed by 2x compared to traditional VLA architectures.