ConLA: Contrastive Latent Action Learning from Human Videos for Robotic Manipulation

TL;DR

ConLA is an unsupervised framework that learns disentangled, semantically meaningful latent actions from human videos, enabling scalable robotic policy pretraining that surpasses real robot data in performance.

Contribution

Introducing ConLA, a contrastive disentanglement method that leverages action priors and temporal cues to improve latent action learning from human videos for robotics.

Findings

ConLA outperforms previous methods on multiple benchmarks.

Pretraining on human videos alone surpasses real robot trajectory pretraining.

The method effectively isolates motion dynamics from visual content.

Abstract

Vision-Language-Action (VLA) models achieve preliminary generalization through pretraining on large scale robot teleoperation datasets. However, acquiring datasets that comprehensively cover diverse tasks and environments is extremely costly and difficult to scale. In contrast, human demonstration videos offer a rich and scalable source of diverse scenes and manipulation behaviors, yet their lack of explicit action supervision hinders direct utilization. Prior work leverages VQ-VAE based frameworks to learn latent actions from human videos in an unsupervised manner. Nevertheless, since the training objective primarily focuses on reconstructing visual appearances rather than capturing inter-frame dynamics, the learned representations tend to rely on spurious visual cues, leading to shortcut learning and entangled latent representations that hinder transferability. To address this, we propose ConLA, an unsupervised pretraining framework for learning robotic policies from human videos. ConLA introduces a contrastive disentanglement mechanism that leverages action category priors and temporal cues to isolate motion dynamics from visual content, effectively mitigating shortcut learning. Extensive experiments show that ConLA achieves strong performance across diverse benchmarks. Notably, by pretraining solely on human videos, our method for the first time surpasses the performance obtained with real robot trajectory pretraining, highlighting its ability to extract pure and semantically consistent latent action representations for scalable robot learning.