Intern-GS: Vision Model Guided Sparse-View 3D Gaussian Splatting

TL;DR

Intern-GS leverages vision foundation models to guide sparse-view 3D Gaussian Splatting, significantly improving scene reconstruction quality by addressing data limitations through enhanced initialization and optimization.

Contribution

The paper introduces Intern-GS, a novel method that integrates vision foundation models into sparse-view 3D Gaussian Splatting for superior scene reconstruction.

Findings

Achieves state-of-the-art rendering quality on multiple datasets.

Effectively alleviates sparse-view limitations with foundation model guidance.

Improves initialization and optimization in 3D Gaussian Splatting.

Abstract

Sparse-view scene reconstruction often faces significant challenges due to the constraints imposed by limited observational data. These limitations result in incomplete information, leading to suboptimal reconstructions using existing methodologies. To address this, we present Intern-GS, a novel approach that effectively leverages rich prior knowledge from vision foundation models to enhance the process of sparse-view Gaussian Splatting, thereby enabling high-quality scene reconstruction. Specifically, Intern-GS utilizes vision foundation models to guide both the initialization and the optimization process of 3D Gaussian splatting, effectively addressing the limitations of sparse inputs. In the initialization process, our method employs DUSt3R to generate a dense and non-redundant gaussian point cloud. This approach significantly alleviates the limitations encountered by traditional structure-from-motion (SfM) methods, which often struggle under sparse-view constraints. During the optimization process, vision foundation models predict depth and appearance for unobserved views, refining the 3D Gaussians to compensate for missing information in unseen regions. Extensive experiments demonstrate that Intern-GS achieves state-of-the-art rendering quality across diverse datasets, including both forward-facing and large-scale scenes, such as LLFF, DTU, and Tanks and Temples.