Self-Ensembling Gaussian Splatting for Few-Shot Novel View Synthesis

TL;DR

This paper introduces Self-Ensembling Gaussian Splatting (SE-GS), a novel method that improves 3D Gaussian Splatting for few-shot novel view synthesis by reducing overfitting through an uncertainty-aware ensemble approach, leading to better generalization.

Contribution

We propose a self-ensembling framework with uncertainty-aware perturbations to enhance 3D Gaussian Splatting's generalization in few-shot NVS scenarios.

Findings

Outperforms state-of-the-art methods on multiple datasets

Enhances NVS quality with limited training views

Maintains computational efficiency during training

Abstract

3D Gaussian Splatting (3DGS) has demonstrated remarkable effectiveness in novel view synthesis (NVS). However, 3DGS tends to overfit when trained with sparse views, limiting its generalization to novel viewpoints. In this paper, we address this overfitting issue by introducing Self-Ensembling Gaussian Splatting (SE-GS). We achieve self-ensembling by incorporating an uncertainty-aware perturbation strategy during training. A $\mathbf{\Delta}$-model and a $\mathbf{\Sigma}$-model are jointly trained on the available images. The $\mathbf{\Delta}$-model is dynamically perturbed based on rendering uncertainty across training steps, generating diverse perturbed models with negligible computational overhead. Discrepancies between the $\mathbf{\Sigma}$-model and these perturbed models are minimized throughout training, forming a robust ensemble of 3DGS models. This ensemble, represented by the $\mathbf{\Sigma}$-model, is then used to generate novel-view images during inference. Experimental results on the LLFF, Mip-NeRF360, DTU, and MVImgNet datasets demonstrate that our approach enhances NVS quality under few-shot training conditions, outperforming existing state-of-the-art methods. The code is released at: https://sailor-z.github.io/projects/SEGS.html.