ViSA-Flow: Accelerating Robot Skill Learning via Large-Scale Video Semantic Action Flow

TL;DR

ViSA-Flow introduces a self-supervised framework that learns semantic action flows from large-scale human videos and adapts this knowledge to robots, enabling efficient skill learning with minimal robot demonstrations.

Contribution

The paper presents a novel semantic action flow representation and a transfer learning framework that leverages large-scale human videos for robot skill acquisition.

Findings

State-of-the-art performance on CALVIN benchmark

Effective transfer from human videos to robots

Strong results in low-data regimes

Abstract

One of the central challenges preventing robots from acquiring complex manipulation skills is the prohibitive cost of collecting large-scale robot demonstrations. In contrast, humans are able to learn efficiently by watching others interact with their environment. To bridge this gap, we introduce semantic action flow as a core intermediate representation capturing the essential spatio-temporal manipulator-object interactions, invariant to superficial visual differences. We present ViSA-Flow, a framework that learns this representation self-supervised from unlabeled large-scale video data. First, a generative model is pre-trained on semantic action flows automatically extracted from large-scale human-object interaction video data, learning a robust prior over manipulation structure. Second, this prior is efficiently adapted to a target robot by fine-tuning on a small set of robot demonstrations processed through the same semantic abstraction pipeline. We demonstrate through extensive experiments on the CALVIN benchmark and real-world tasks that ViSA-Flow achieves state-of-the-art performance, particularly in low-data regimes, outperforming prior methods by effectively transferring knowledge from human video observation to robotic execution. Videos are available at https://visaflow-web.github.io/ViSAFLOW.