X-Sim: Cross-Embodiment Learning via Real-to-Sim-to-Real

TL;DR

X-Sim introduces a novel cross-embodiment learning framework that leverages object motion from human videos to train and transfer robot manipulation policies without requiring robot teleoperation data.

Contribution

It proposes a real-to-sim-to-real approach using object-centric rewards, synthetic diffusion policies, and online domain adaptation for effective robot policy learning from human videos.

Findings

Improves task progress by 30% over baselines

Matches behavior cloning with 10x less data

Generalizes to new viewpoints and test conditions

Abstract

Human videos offer a scalable way to train robot manipulation policies, but lack the action labels needed by standard imitation learning algorithms. Existing cross-embodiment approaches try to map human motion to robot actions, but often fail when the embodiments differ significantly. We propose X-Sim, a real-to-sim-to-real framework that uses object motion as a dense and transferable signal for learning robot policies. X-Sim starts by reconstructing a photorealistic simulation from an RGBD human video and tracking object trajectories to define object-centric rewards. These rewards are used to train a reinforcement learning (RL) policy in simulation. The learned policy is then distilled into an image-conditioned diffusion policy using synthetic rollouts rendered with varied viewpoints and lighting. To transfer to the real world, X-Sim introduces an online domain adaptation technique that aligns real and simulated observations during deployment. Importantly, X-Sim does not require any robot teleoperation data. We evaluate it across 5 manipulation tasks in 2 environments and show that it: (1) improves task progress by 30% on average over hand-tracking and sim-to-real baselines, (2) matches behavior cloning with 10x less data collection time, and (3) generalizes to new camera viewpoints and test-time changes. Code and videos are available at https://portal-cornell.github.io/X-Sim/.