LM-Gaussian: Boost Sparse-view 3D Gaussian Splatting with Large Model Priors

TL;DR

LM-Gaussian leverages large-scale vision model priors to enable high-quality 3D scene reconstruction from sparse images, reducing data needs and improving detail preservation.

Contribution

The paper introduces LM-Gaussian, a novel method that uses stereo, diffusion, and video priors to enhance sparse-view 3D Gaussian Splatting reconstructions.

Findings

Achieves high-quality 3D reconstructions with fewer images.

Outperforms existing methods on public datasets.

Preserves scene details effectively during sparse-view reconstruction.

Abstract

We aim to address sparse-view reconstruction of a 3D scene by leveraging priors from large-scale vision models. While recent advancements such as 3D Gaussian Splatting (3DGS) have demonstrated remarkable successes in 3D reconstruction, these methods typically necessitate hundreds of input images that densely capture the underlying scene, making them time-consuming and impractical for real-world applications. However, sparse-view reconstruction is inherently ill-posed and under-constrained, often resulting in inferior and incomplete outcomes. This is due to issues such as failed initialization, overfitting on input images, and a lack of details. To mitigate these challenges, we introduce LM-Gaussian, a method capable of generating high-quality reconstructions from a limited number of images. Specifically, we propose a robust initialization module that leverages stereo priors to aid in the recovery of camera poses and the reliable point clouds. Additionally, a diffusion-based refinement is iteratively applied to incorporate image diffusion priors into the Gaussian optimization process to preserve intricate scene details. Finally, we utilize video diffusion priors to further enhance the rendered images for realistic visual effects. Overall, our approach significantly reduces the data acquisition requirements compared to previous 3DGS methods. We validate the effectiveness of our framework through experiments on various public datasets, demonstrating its potential for high-quality 360-degree scene reconstruction. Visual results are on our website.