AMPLIFY: Actionless Motion Priors for Robot Learning from Videos

TL;DR

AMPLIFY introduces a modular framework that learns accurate, generalizable robot dynamics from large-scale action-free videos and limited action-labeled data, significantly improving policy learning and video prediction.

Contribution

It decouples visual motion prediction from action inference, enabling scalable learning of dynamics from heterogeneous data sources for robotics.

Findings

Achieves up to 3.7x better MSE in dynamics prediction.

Enables 1.2-2.2x improvement in low-data policy learning.

Generalizes to LIBERO tasks from zero in-distribution action data.

Abstract

Action-labeled data for robotics is scarce and expensive, limiting the generalization of learned policies. In contrast, vast amounts of action-free video data are readily available, but translating these observations into effective policies remains a challenge. We introduce AMPLIFY, a novel framework that leverages large-scale video data by encoding visual dynamics into compact, discrete motion tokens derived from keypoint trajectories. Our modular approach separates visual motion prediction from action inference, decoupling the challenges of learning what motion defines a task from how robots can perform it. We train a forward dynamics model on abundant action-free videos and an inverse dynamics model on a limited set of action-labeled examples, allowing for independent scaling. Extensive evaluations demonstrate that the learned dynamics are both accurate, achieving up to 3.7x better MSE and over 2.5x better pixel prediction accuracy compared to prior approaches, and broadly useful. In downstream policy learning, our dynamics predictions enable a 1.2-2.2x improvement in low-data regimes, a 1.4x average improvement by learning from action-free human videos, and the first generalization to LIBERO tasks from zero in-distribution action data. Beyond robotic control, we find the dynamics learned by AMPLIFY to be a versatile latent world model, enhancing video prediction quality. Our results present a novel paradigm leveraging heterogeneous data sources to build efficient, generalizable world models. More information can be found at https://amplify-robotics.github.io/.