LiteReality: Graphics-Ready 3D Scene Reconstruction from RGB-D Scans

TL;DR

LiteReality is a pipeline that converts RGB-D indoor scans into compact, realistic, and interactive 3D scenes suitable for graphics applications, supporting object individuality, articulation, high-quality materials, and physical interaction.

Contribution

It introduces a novel, training-free object retrieval and material painting approach that enhances realism and compatibility with graphics pipelines, advancing scene reconstruction from RGB-D data.

Findings

Achieves state-of-the-art similarity performance on Scan2CAD benchmark.

Produces scenes compatible with standard graphics pipelines for AR/VR, gaming, and robotics.

Demonstrates robustness under severe misalignment, occlusion, and poor lighting conditions.

Abstract

We propose LiteReality, a novel pipeline that converts RGB-D scans of indoor environments into compact, realistic, and interactive 3D virtual replicas. LiteReality not only reconstructs scenes that visually resemble reality but also supports key features essential for graphics pipelines -- such as object individuality, articulation, high-quality physically based rendering materials, and physically based interaction. At its core, LiteReality first performs scene understanding and parses the results into a coherent 3D layout and objects with the help of a structured scene graph. It then reconstructs the scene by retrieving the most visually similar 3D artist-crafted models from a curated asset database. Next, the Material Painting module enhances realism by recovering high-quality, spatially varying materials. Finally, the reconstructed scene is integrated into a simulation engine with basic physical properties to enable interactive behavior. The resulting scenes are compact, editable, and fully compatible with standard graphics pipelines, making them suitable for applications in AR/VR, gaming, robotics, and digital twins. In addition, LiteReality introduces a training-free object retrieval module that achieves state-of-the-art similarity performance on the Scan2CAD benchmark, along with a robust material painting module capable of transferring appearances from images of any style to 3D assets -- even under severe misalignment, occlusion, and poor lighting. We demonstrate the effectiveness of LiteReality on both real-life scans and public datasets. Project page: https://litereality.github.io; Video: https://www.youtube.com/watch?v=ecK9m3LXg2c