HAND Me the Data: Fast Robot Adaptation via Hand Path Retrieval

TL;DR

This paper introduces HAND, a quick and efficient method for robot learning from human hand demonstrations, enabling rapid task adaptation without complex calibration or detailed pose estimation.

Contribution

HAND leverages visual hand tracking and behavior retrieval from robot play data to enable fast, calibration-free robot task learning from human demonstrations.

Findings

HAND achieves over 2x higher success rates than baselines.

Fine-tuning on retrieved data enables real-time task learning in under four minutes.

The method does not require calibrated cameras or detailed hand pose estimation.

Abstract

We hand the community HAND, a simple and time-efficient method for teaching robots new manipulation tasks through human hand demonstrations. Instead of relying on task-specific robot demonstrations collected via teleoperation, HAND uses easy-to-provide hand demonstrations to retrieve relevant behaviors from task-agnostic robot play data. Using a visual tracking pipeline, HAND extracts the motion of the human hand from the hand demonstration and retrieves robot sub-trajectories in two stages: first filtering by visual similarity, then retrieving trajectories with similar behaviors to the hand. Fine-tuning a policy on the retrieved data enables real-time learning of tasks in under four minutes, without requiring calibrated cameras or detailed hand pose estimation. Experiments also show that HAND outperforms retrieval baselines by over 2x in average task success rates on real robots. Videos can be found at our project website: https://liralab.usc.edu/handretrieval/.