Lucid-XR: An Extended-Reality Data Engine for Robotic Manipulation

TL;DR

Lucid-XR is a novel extended-reality data engine that generates diverse, realistic multi-modal data for training robotic systems, enabling zero-shot transfer in complex environments.

Contribution

It introduces a web-based physics simulation environment integrated with human-to-robot pose retargeting and a physics-guided video generation pipeline for robotic training data.

Findings

Zero-shot transfer of visual policies to unseen environments

Synthetic data enables effective training for complex manipulation tasks

System operates directly on XR headsets for immersive data collection

Abstract

We introduce Lucid-XR, a generative data engine for creating diverse and realistic-looking multi-modal data to train real-world robotic systems. At the core of Lucid-XR is vuer, a web-based physics simulation environment that runs directly on the XR headset, enabling internet-scale access to immersive, latency-free virtual interactions without requiring specialized equipment. The complete system integrates on-device physics simulation with human-to-robot pose retargeting. Data collected is further amplified by a physics-guided video generation pipeline steerable via natural language specifications. We demonstrate zero-shot transfer of robot visual policies to unseen, cluttered, and badly lit evaluation environments, after training entirely on Lucid-XR's synthetic data. We include examples across dexterous manipulation tasks that involve soft materials, loosely bound particles, and rigid body contact. Project website: https://lucidxr.github.io