ComSim: Building Scalable Real-World Robot Data Generation via Compositional Simulation

TL;DR

ComSim introduces a hybrid simulation method combining classical and neural simulation to generate diverse, real-world consistent training data, significantly reducing the sim2real gap in robotics.

Contribution

The paper presents Compositional Simulation, a novel hybrid approach that enhances data generation for robotics by integrating classical and neural simulation techniques.

Findings

Reduces the sim2real domain gap significantly.

Improves success rates of real-world policy training.

Generates large-scale, diverse training datasets.

Abstract

Recent advancements in foundational models, such as large language models and world models, have greatly enhanced the capabilities of robotics, enabling robots to autonomously perform complex tasks. However, acquiring large-scale, high-quality training data for robotics remains a challenge, as it often requires substantial manual effort and is limited in its coverage of diverse real-world environments. To address this, we propose a novel hybrid approach called Compositional Simulation, which combines classical simulation and neural simulation to generate accurate action-video pairs while maintaining real-world consistency. Our approach utilizes a closed-loop real-sim-real data augmentation pipeline, leveraging a small amount of real-world data to generate diverse, large-scale training datasets that cover a broader spectrum of real-world scenarios. We train a neural simulator to transform classical simulation videos into real-world representations, improving the accuracy of policy models trained in real-world environments. Through extensive experiments, we demonstrate that our method significantly reduces the sim2real domain gap, resulting in higher success rates in real-world policy model training. Our approach offers a scalable solution for generating robust training data and bridging the gap between simulated and real-world robotics.