TraceGen: World Modeling in 3D Trace Space Enables Learning from Cross-Embodiment Videos

TL;DR

TraceGen introduces a 3D trace-space world model that enables robots to learn new tasks from cross-embodiment videos efficiently, significantly reducing data requirements and inference time.

Contribution

The paper presents TraceGen, a novel world model that predicts motion in a symbolic 3D trace-space, facilitating cross-embodiment learning from heterogeneous videos.

Findings

Achieves 80% success with five target videos across four tasks.

Offers 50-600x faster inference than pixel-based models.

Reaches 67.5% success with uncalibrated human videos on real robots.

Abstract

Learning new robot tasks on new platforms and in new scenes from only a handful of demonstrations remains challenging. While videos of other embodiments - humans and different robots - are abundant, differences in embodiment, camera, and environment hinder their direct use. We address the small-data problem by introducing a unifying, symbolic representation - a compact 3D "trace-space" of scene-level trajectories - that enables learning from cross-embodiment, cross-environment, and cross-task videos. We present TraceGen, a world model that predicts future motion in trace-space rather than pixel space, abstracting away appearance while retaining the geometric structure needed for manipulation. To train TraceGen at scale, we develop TraceForge, a data pipeline that transforms heterogeneous human and robot videos into consistent 3D traces, yielding a corpus of 123K videos and 1.8M observation-trace-language triplets. Pretraining on this corpus produces a transferable 3D motion prior that adapts efficiently: with just five target robot videos, TraceGen attains 80% success across four tasks while offering 50-600x faster inference than state-of-the-art video-based world models. In the more challenging case where only five uncalibrated human demonstration videos captured on a handheld phone are available, it still reaches 67.5% success on a real robot, highlighting TraceGen's ability to adapt across embodiments without relying on object detectors or heavy pixel-space generation.