Integrating Meshes and 3D Gaussians for Indoor Scene Reconstruction with SAM Mask Guidance

TL;DR

This paper introduces a hybrid 3D scene reconstruction method combining meshes for room layouts and 3D Gaussians for objects, guided by SAM masks for improved training stability and quality.

Contribution

The novel integration of meshes and 3D Gaussians with SAM guidance and a densification stage enhances indoor scene reconstruction and editing capabilities.

Findings

Improved separation of scene primitives during training.

Enhanced reconstruction quality with the densification stage.

Stable joint training of meshes and Gaussians achieved.

Abstract

We present a novel approach for 3D indoor scene reconstruction that combines 3D Gaussian Splatting (3DGS) with mesh representations. We use meshes for the room layout of the indoor scene, such as walls, ceilings, and floors, while employing 3D Gaussians for other objects. This hybrid approach leverages the strengths of both representations, offering enhanced flexibility and ease of editing. However, joint training of meshes and 3D Gaussians is challenging because it is not clear which primitive should affect which part of the rendered image. Objects close to the room layout often struggle during training, particularly when the room layout is textureless, which can lead to incorrect optimizations and unnecessary 3D Gaussians. To overcome these challenges, we employ Segment Anything Model (SAM) to guide the selection of primitives. The SAM mask loss enforces each instance to be represented by either Gaussians or meshes, ensuring clear separation and stable training. Furthermore, we introduce an additional densification stage without resetting the opacity after the standard densification. This stage mitigates the degradation of image quality caused by a limited number of 3D Gaussians after the standard densification.