CAST: Component-Aligned 3D Scene Reconstruction from an RGB Image

TL;DR

CAST is a novel method that reconstructs high-quality 3D scenes from a single RGB image by integrating object segmentation, spatial relationship analysis, occlusion-aware generation, and physics-based correction for realistic and coherent scene modeling.

Contribution

The paper introduces CAST, a comprehensive framework combining GPT-based analysis, large-scale 3D generation, and physics-aware optimization for improved 3D scene reconstruction from a single image.

Findings

Effective occlusion handling with Signed Distance Fields

Accurate object alignment and scene coherence

Enhanced physical realism in reconstructed scenes

Abstract

Recovering high-quality 3D scenes from a single RGB image is a challenging task in computer graphics. Current methods often struggle with domain-specific limitations or low-quality object generation. To address these, we propose CAST (Component-Aligned 3D Scene Reconstruction from a Single RGB Image), a novel method for 3D scene reconstruction and recovery. CAST starts by extracting object-level 2D segmentation and relative depth information from the input image, followed by using a GPT-based model to analyze inter-object spatial relationships. This enables the understanding of how objects relate to each other within the scene, ensuring more coherent reconstruction. CAST then employs an occlusion-aware large-scale 3D generation model to independently generate each object's full geometry, using MAE and point cloud conditioning to mitigate the effects of occlusions and partial object information, ensuring accurate alignment with the source image's geometry and texture. To align each object with the scene, the alignment generation model computes the necessary transformations, allowing the generated meshes to be accurately placed and integrated into the scene's point cloud. Finally, CAST incorporates a physics-aware correction step that leverages a fine-grained relation graph to generate a constraint graph. This graph guides the optimization of object poses, ensuring physical consistency and spatial coherence. By utilizing Signed Distance Fields (SDF), the model effectively addresses issues such as occlusions, object penetration, and floating objects, ensuring that the generated scene accurately reflects real-world physical interactions. CAST can be leveraged in robotics, enabling efficient real-to-simulation workflows and providing realistic, scalable simulation environments for robotic systems.