NovelGS: Consistent Novel-view Denoising via Large Gaussian Reconstruction Model

TL;DR

NovelGS introduces a transformer-based diffusion model for Gaussian Splatting that improves 3D reconstruction from sparse views, producing consistent, sharp textures and surpassing existing methods in quality and robustness.

Contribution

The paper presents a novel transformer-based denoising approach for Gaussian Splatting that effectively reconstructs 3D objects with unseen regions, outperforming prior feed-forward methods.

Findings

State-of-the-art multi-view reconstruction performance.

Effective generative modeling of unseen regions.

Superior qualitative and quantitative results on datasets.

Abstract

We introduce NovelGS, a diffusion model for Gaussian Splatting (GS) given sparse-view images. Recent works leverage feed-forward networks to generate pixel-aligned Gaussians, which could be fast rendered. Unfortunately, the method was unable to produce satisfactory results for areas not covered by the input images due to the formulation of these methods. In contrast, we leverage the novel view denoising through a transformer-based network to generate 3D Gaussians. Specifically, by incorporating both conditional views and noisy target views, the network predicts pixel-aligned Gaussians for each view. During training, the rendered target and some additional views of the Gaussians are supervised. During inference, the target views are iteratively rendered and denoised from pure noise. Our approach demonstrates state-of-the-art performance in addressing the multi-view image reconstruction challenge. Due to generative modeling of unseen regions, NovelGS effectively reconstructs 3D objects with consistent and sharp textures. Experimental results on publicly available datasets indicate that NovelGS substantially surpasses existing image-to-3D frameworks, both qualitatively and quantitatively. We also demonstrate the potential of NovelGS in generative tasks, such as text-to-3D and image-to-3D, by integrating it with existing multiview diffusion models. We will make the code publicly accessible.