Learning from Watching: Scalable Extraction of Manipulation Trajectories from Human Videos

TL;DR

This paper introduces a scalable method that leverages foundation models and point tracking to extract detailed manipulation trajectories from online human videos, enhancing data efficiency for robotic learning.

Contribution

It combines foundation models with point tracking to fully utilize human videos for robotic manipulation data extraction, surpassing previous focus on hand detection or object pose estimation.

Findings

Accurately tracks keypoints during manipulation tasks

Enables scalable extraction of manipulation trajectories from videos

Improves data efficiency for robot learning

Abstract

Collecting high-quality data for training large-scale robotic models typically relies on real robot platforms, which is labor-intensive and costly, whether via teleoperation or scripted demonstrations. To scale data collection, many researchers have turned to leveraging human manipulation videos available online. However, current methods predominantly focus on hand detection or object pose estimation, failing to fully exploit the rich interaction cues embedded in these videos. In this work, we propose a novel approach that combines large foundation models for video understanding with point tracking techniques to extract dense trajectories of all task-relevant keypoints during manipulation. This enables more comprehensive utilization of Internet-scale human demonstration videos. Experimental results demonstrate that our method can accurately track keypoints throughout the entire manipulation process, paving the way for more scalable and data-efficient robot learning.