AugGS: Self-augmented Gaussians with Structural Masks for Sparse-view 3D Reconstruction

TL;DR

AugGS introduces a self-augmented Gaussian splatting framework with structural masks that significantly improves sparse-view 3D reconstruction by combining initial Gaussian modeling with diffusion-based image enhancement.

Contribution

The paper presents a novel two-stage Gaussian splatting method with structural masking and diffusion model fine-tuning for enhanced sparse-view 3D reconstruction.

Findings

Achieves state-of-the-art results on benchmark datasets.

Improves perceptual quality and multi-view consistency.

Enhances robustness to sparse inputs and noise.

Abstract

Sparse-view 3D reconstruction is a major challenge in computer vision, aiming to create complete three-dimensional models from limited viewing angles. Key obstacles include: 1) a small number of input images with inconsistent information; 2) dependence on input image quality; and 3) large model parameter sizes. To tackle these issues, we propose a self-augmented two-stage Gaussian splatting framework enhanced with structural masks for sparse-view 3D reconstruction. Initially, our method generates a basic 3D Gaussian representation from sparse inputs and renders multi-view images. We then fine-tune a pre-trained 2D diffusion model to enhance these images, using them as augmented data to further optimize the 3D Gaussians. Additionally, a structural masking strategy during training enhances the model's robustness to sparse inputs and noise. Experiments on benchmarks like MipNeRF360, OmniObject3D, and OpenIllumination demonstrate that our approach achieves state-of-the-art performance in perceptual quality and multi-view consistency with sparse inputs.