HDRSplat: Gaussian Splatting for High Dynamic Range 3D Scene Reconstruction from Raw Images

TL;DR

HDRSplat introduces a novel 3D Gaussian Splatting method that directly trains on 14-bit raw images, enabling fast, high-fidelity HDR scene reconstruction in challenging lighting conditions, surpassing prior methods in speed and accuracy.

Contribution

The paper presents a new HDR-compatible 3D Gaussian Splatting approach with a linear HDR loss and rasterization tuning, significantly improving reconstruction quality and speed in high dynamic range scenes.

Findings

Reconstructs HDR scenes in under 15 minutes per scene.

Achieves inference speeds of over 120 fps.

Outperforms prior methods like RawNeRF in speed and quality.

Abstract

The recent advent of 3D Gaussian Splatting (3DGS) has revolutionized the 3D scene reconstruction space enabling high-fidelity novel view synthesis in real-time. However, with the exception of RawNeRF, all prior 3DGS and NeRF-based methods rely on 8-bit tone-mapped Low Dynamic Range (LDR) images for scene reconstruction. Such methods struggle to achieve accurate reconstructions in scenes that require a higher dynamic range. Examples include scenes captured in nighttime or poorly lit indoor spaces having a low signal-to-noise ratio, as well as daylight scenes with shadow regions exhibiting extreme contrast. Our proposed method HDRSplat tailors 3DGS to train directly on 14-bit linear raw images in near darkness which preserves the scenes' full dynamic range and content. Our key contributions are two-fold: Firstly, we propose a linear HDR space-suited loss that effectively extracts scene information from noisy dark regions and nearly saturated bright regions simultaneously, while also handling view-dependent colors without increasing the degree of spherical harmonics. Secondly, through careful rasterization tuning, we implicitly overcome the heavy reliance and sensitivity of 3DGS on point cloud initialization. This is critical for accurate reconstruction in regions of low texture, high depth of field, and low illumination. HDRSplat is the fastest method to date that does 14-bit (HDR) 3D scene reconstruction in $\le$15 minutes/scene ($\sim$30x faster than prior state-of-the-art RawNeRF). It also boasts the fastest inference speed at $\ge$120fps. We further demonstrate the applicability of our HDR scene reconstruction by showcasing various applications like synthetic defocus, dense depth map extraction, and post-capture control of exposure, tone-mapping and view-point.