MetaScenes: Towards Automated Replica Creation for Real-world 3D Scans

TL;DR

MetaScenes is a large-scale dataset of real-world 3D scans combined with Scan2Sim, a model for automated asset replacement, enabling scalable, realistic scene creation for embodied AI research.

Contribution

The paper introduces MetaScenes, a comprehensive 3D scene dataset from real-world scans, and Scan2Sim, a multi-modal alignment model for automated scene asset replacement, reducing reliance on manual artist-driven design.

Findings

MetaScenes includes 15,366 objects across 831 categories.

Scan2Sim achieves high-quality, automated asset replacement.

Benchmarks demonstrate improved generalization in embodied AI tasks.

Abstract

Embodied AI (EAI) research requires high-quality, diverse 3D scenes to effectively support skill acquisition, sim-to-real transfer, and generalization. Achieving these quality standards, however, necessitates the precise replication of real-world object diversity. Existing datasets demonstrate that this process heavily relies on artist-driven designs, which demand substantial human effort and present significant scalability challenges. To scalably produce realistic and interactive 3D scenes, we first present MetaScenes, a large-scale, simulatable 3D scene dataset constructed from real-world scans, which includes 15366 objects spanning 831 fine-grained categories. Then, we introduce Scan2Sim, a robust multi-modal alignment model, which enables the automated, high-quality replacement of assets, thereby eliminating the reliance on artist-driven designs for scaling 3D scenes. We further propose two benchmarks to evaluate MetaScenes: a detailed scene synthesis task focused on small item layouts for robotic manipulation and a domain transfer task in vision-and-language navigation (VLN) to validate cross-domain transfer. Results confirm MetaScene's potential to enhance EAI by supporting more generalizable agent learning and sim-to-real applications, introducing new possibilities for EAI research. Project website: https://meta-scenes.github.io/.