S-VAM: Shortcut Video-Action Model by Self-Distilling Geometric and Semantic Foresight

TL;DR

S-VAM introduces a real-time, high-fidelity video action prediction model that uses self-distillation to efficiently generate coherent geometric and semantic foresight from a single forward pass, improving robot manipulation tasks.

Contribution

The paper proposes a novel shortcut video-action model with a self-distillation strategy that condenses multi-step generative priors into one-step inference, enabling efficient and accurate action prediction.

Findings

Outperforms state-of-the-art methods in simulation and real-world tasks.

Enables real-time inference with high-fidelity foresight.

Improves robotic manipulation in complex environments.

Abstract

Video action models (VAMs) have emerged as a promising paradigm for robot learning, owing to their powerful visual foresight for complex manipulation tasks. However, current VAMs, typically relying on either slow multi-step video generation or noisy one-step feature extraction, cannot simultaneously guarantee real-time inference and high-fidelity foresight. To address this limitation, we propose S-VAM, a shortcut video-action model that foresees coherent geometric and semantic representations via a single forward pass. Serving as a stable blueprint, these foreseen representations significantly simplify the action prediction. To enable this efficient shortcut, we introduce a novel self-distillation strategy that condenses structured generative priors of multi-step denoising into one-step inference. Specifically, vision foundation model (VFM) representations extracted from the diffusion model's own multi-step generated videos provide teacher targets. Lightweight decouplers, as students, learn to directly map noisy one-step features to these targets. Extensive experiments in simulation and the real world demonstrate that our S-VAM outperforms state-of-the-art methods, enabling efficient and precise manipulation in complex environments. Our project page is https://haodong-yan.github.io/S-VAM/