OKAMI: Teaching Humanoid Robots Manipulation Skills through Single Video Imitation

TL;DR

OKAMI introduces a novel method for teaching humanoid robots manipulation skills by imitating from a single RGB-D video, enabling generalization and policy training without extensive manual effort.

Contribution

The paper presents a new approach combining object-aware retargeting and open-world vision models for single-video imitation learning in humanoid robots.

Findings

Outperforms state-of-the-art in open-world imitation

Achieves 79.2% success rate in visuomotor policy execution

Enables generalization across diverse visual and spatial conditions

Abstract

We study the problem of teaching humanoid robots manipulation skills by imitating from single video demonstrations. We introduce OKAMI, a method that generates a manipulation plan from a single RGB-D video and derives a policy for execution. At the heart of our approach is object-aware retargeting, which enables the humanoid robot to mimic the human motions in an RGB-D video while adjusting to different object locations during deployment. OKAMI uses open-world vision models to identify task-relevant objects and retarget the body motions and hand poses separately. Our experiments show that OKAMI achieves strong generalizations across varying visual and spatial conditions, outperforming the state-of-the-art baseline on open-world imitation from observation. Furthermore, OKAMI rollout trajectories are leveraged to train closed-loop visuomotor policies, which achieve an average success rate of 79.2% without the need for labor-intensive teleoperation. More videos can be found on our website https://ut-austin-rpl.github.io/OKAMI/.