Learning Skills from Action-Free Videos

TL;DR

This paper introduces SOF, a framework that learns high-level skills from action-free videos using optical flow, enabling better planning and execution of complex robot actions from visual data.

Contribution

It proposes a novel flow-based latent skill space that bridges the gap between video prediction and action translation, facilitating high-level planning from raw videos.

Findings

Improves multitask learning performance

Enables long-horizon skill composition

Learns directly from raw visual data

Abstract

Learning from videos offers a promising path toward generalist robots by providing rich visual and temporal priors beyond what real robot datasets contain. While existing video generative models produce impressive visual predictions, they are difficult to translate into low-level actions. Conversely, latent-action models better align videos with actions, but they typically operate at the single-step level and lack high-level planning capabilities. We bridge this gap by introducing Skill Abstraction from Optical Flow (SOF), a framework that learns latent skills from large collections of action-free videos. Our key idea is to learn a latent skill space through an intermediate representation based on optical flow that captures motion information aligned with both video dynamics and robot actions. By learning skills in this flow-based latent space, SOF enables high-level planning over video-derived skills and allows for easier translation of these skills into actions. Experiments show that our approach consistently improves performance in both multitask and long-horizon settings, demonstrating the ability to acquire and compose skills directly from raw visual data.