From Seeing to Simulating: Generative High-Fidelity Simulation with Digital Cousins for Generalizable Robot Learning and Evaluation

TL;DR

This paper introduces a generative framework that converts real-world panoramas into high-fidelity simulation scenes, enabling scalable data augmentation for improved robot learning and evaluation.

Contribution

A novel real-to-sim generative approach that creates diverse, high-quality simulation environments from real-world data, supporting complex manipulation and navigation tasks.

Findings

Generated scenes show strong sim-to-real correlation.

Scaling data generation improves generalization to unseen environments.

The platform supports interactive manipulation and long-horizon navigation.

Abstract

Learning robust robot policies in real-world environments requires diverse data augmentation, yet scaling real-world data collection is costly due to the need for acquiring physical assets and reconfiguring environments. Therefore, augmenting real-world scenes into simulation has become a practical augmentation for efficient learning and evaluation. We present a generative framework that establishes a generative real-to-sim mapping from real-world panoramas to high-fidelity simulation scenes, and further synthesize diverse cousin scenes via semantic and geometric editing. Combined with high-quality physics engines and realistic assets, the generated scenes support interactive manipulation tasks. Additionally, we incorporate multi-room stitching to construct consistent large-scale environments for long-horizon navigation across complex layouts. Experiments demonstrate a strong sim-to-real correlation validating our platform's fidelity, and show that extensively scaling up data generation leads to significantly better generalization to unseen scene and object variations, demonstrating the effectiveness of Digital Cousins for generalizable robot learning and evaluation.