One-Shot Imitation under Mismatched Execution

TL;DR

RHyME is a novel framework that enables robots to imitate human demonstrations by automatically pairing and synthesizing human videos from robot trajectories, overcoming data pairing and visual similarity challenges.

Contribution

The paper introduces RHyME, a sequence-level optimal transport-based method for cross-embodiment imitation that does not require paired data or frame-level visual similarity.

Findings

Achieves over 50% increase in task success rate.

Successfully imitates cross-embodiment demonstrations in simulation and real-world.

Facilitates policy training without paired human-robot data.

Abstract

Human demonstrations as prompts are a powerful way to program robots to do long-horizon manipulation tasks. However, translating these demonstrations into robot-executable actions presents significant challenges due to execution mismatches in movement styles and physical capabilities. Existing methods for human-robot translation either depend on paired data, which is infeasible to scale, or rely heavily on frame-level visual similarities that often break down in practice. To address these challenges, we propose RHyME, a novel framework that automatically pairs human and robot trajectories using sequence-level optimal transport cost functions. Given long-horizon robot demonstrations, RHyME synthesizes semantically equivalent human videos by retrieving and composing short-horizon human clips. This approach facilitates effective policy training without the need for paired data. RHyME successfully imitates a range of cross-embodiment demonstrators, both in simulation and with a real human hand, achieving over 50% increase in task success compared to previous methods. We release our code and datasets at https://portal-cornell.github.io/rhyme/.