MimicFunc: Imitating Tool Manipulation from a Single Human Video via Functional Correspondence

TL;DR

MimicFunc enables robots to learn tool manipulation skills from a single human video by establishing functional correspondences, allowing generalization to new tools and reducing the need for extensive teleoperation data.

Contribution

The paper introduces MimicFunc, a novel framework that uses a function-centric local coordinate frame to achieve one-shot imitation of tool manipulation from human videos.

Findings

Successfully generalizes manipulation skills to novel tools.

Enables training visuomotor policies without teleoperation data.

Demonstrates effective imitation from a single RGB-D video.

Abstract

Imitating tool manipulation from human videos offers an intuitive approach to teaching robots, while also providing a promising and scalable alternative to labor-intensive teleoperation data collection for visuomotor policy learning. While humans can mimic tool manipulation behavior by observing others perform a task just once and effortlessly transfer the skill to diverse tools for functionally equivalent tasks, current robots struggle to achieve this level of generalization. A key challenge lies in establishing function-level correspondences, considering the significant geometric variations among functionally similar tools, referred to as intra-function variations. To address this challenge, we propose MimicFunc, a framework that establishes functional correspondences with function frame, a function-centric local coordinate frame constructed with keypoint-based abstraction, for imitating tool manipulation skills. Experiments demonstrate that MimicFunc effectively enables the robot to generalize the skill from a single RGB-D human video to manipulating novel tools for functionally equivalent tasks. Furthermore, leveraging MimicFunc's one-shot generalization capability, the generated rollouts can be used to train visuomotor policies without requiring labor-intensive teleoperation data collection for novel objects. Our code and video are available at https://sites.google.com/view/mimicfunc.