DIP-GS: Deep Image Prior For Gaussian Splatting Sparse View Recovery

TL;DR

DIP-GS introduces a deep image prior-based method for 3D Gaussian Splatting that effectively reconstructs scenes from sparse views without pre-trained models, outperforming existing approaches.

Contribution

It presents a novel DIP-based 3D Gaussian Splatting approach that enables sparse view scene reconstruction without pre-training, achieving state-of-the-art results.

Findings

State-of-the-art performance on sparse-view reconstruction tasks

Effective scene recovery with limited input views

No reliance on pre-trained models or depth estimation

Abstract

3D Gaussian Splatting (3DGS) is a leading 3D scene reconstruction method, obtaining high-quality reconstruction with real-time rendering runtime performance. The main idea behind 3DGS is to represent the scene as a collection of 3D gaussians, while learning their parameters to fit the given views of the scene. While achieving superior performance in the presence of many views, 3DGS struggles with sparse view reconstruction, where the input views are sparse and do not fully cover the scene and have low overlaps. In this paper, we propose DIP-GS, a Deep Image Prior (DIP) 3DGS representation. By using the DIP prior, which utilizes internal structure and patterns, with coarse-to-fine manner, DIP-based 3DGS can operate in scenarios where vanilla 3DGS fails, such as sparse view recovery. Note that our approach does not use any pre-trained models such as generative models and depth estimation, but rather relies only on the input frames. Among such methods, DIP-GS obtains state-of-the-art (SOTA) competitive results on various sparse-view reconstruction tasks, demonstrating its capabilities.