From Chaos to Clarity: 3DGS in the Dark

TL;DR

This paper introduces a self-supervised learning framework to improve HDR 3D Gaussian Splatting from noisy raw images, enhancing reconstruction quality and speed, especially with limited views.

Contribution

It presents a novel self-supervised approach that incorporates a noise extractor and noise-robust loss to address noise issues in 3DGS from raw images.

Findings

Outperforms existing LDR/HDR 3DGS methods in quality and speed

Effective with limited training views

Demonstrates robustness to raw image noise

Abstract

Novel view synthesis from raw images provides superior high dynamic range (HDR) information compared to reconstructions from low dynamic range RGB images. However, the inherent noise in unprocessed raw images compromises the accuracy of 3D scene representation. Our study reveals that 3D Gaussian Splatting (3DGS) is particularly susceptible to this noise, leading to numerous elongated Gaussian shapes that overfit the noise, thereby significantly degrading reconstruction quality and reducing inference speed, especially in scenarios with limited views. To address these issues, we introduce a novel self-supervised learning framework designed to reconstruct HDR 3DGS from a limited number of noisy raw images. This framework enhances 3DGS by integrating a noise extractor and employing a noise-robust reconstruction loss that leverages a noise distribution prior. Experimental results show that our method outperforms LDR/HDR 3DGS and previous state-of-the-art (SOTA) self-supervised and supervised pre-trained models in both reconstruction quality and inference speed on the RawNeRF dataset across a broad range of training views. Code can be found in \url{https://lizhihao6.github.io/Raw3DGS}.