Lighting Every Darkness with 3DGS: Fast Training and Real-Time Rendering for HDR View Synthesis

TL;DR

LE3D leverages 3D Gaussian Splatting with novel techniques to enable fast training and real-time HDR view synthesis from RAW images, overcoming previous limitations in nighttime scenes and low SNR conditions.

Contribution

The paper introduces LE3D, a method that combines Cone Scatter Initialization, a Color MLP, and regularizations to achieve real-time HDR view synthesis from RAW images using 3DGS.

Findings

Training time reduced to 1% of previous methods

Rendering speed increased by up to 4,000 times

Effective HDR rendering and refocusing in real-time

Abstract

Volumetric rendering based methods, like NeRF, excel in HDR view synthesis from RAWimages, especially for nighttime scenes. While, they suffer from long training times and cannot perform real-time rendering due to dense sampling requirements. The advent of 3D Gaussian Splatting (3DGS) enables real-time rendering and faster training. However, implementing RAW image-based view synthesis directly using 3DGS is challenging due to its inherent drawbacks: 1) in nighttime scenes, extremely low SNR leads to poor structure-from-motion (SfM) estimation in distant views; 2) the limited representation capacity of spherical harmonics (SH) function is unsuitable for RAW linear color space; and 3) inaccurate scene structure hampers downstream tasks such as refocusing. To address these issues, we propose LE3D (Lighting Every darkness with 3DGS). Our method proposes Cone Scatter Initialization to enrich the estimation of SfM, and replaces SH with a Color MLP to represent the RAW linear color space. Additionally, we introduce depth distortion and near-far regularizations to improve the accuracy of scene structure for downstream tasks. These designs enable LE3D to perform real-time novel view synthesis, HDR rendering, refocusing, and tone-mapping changes. Compared to previous volumetric rendering based methods, LE3D reduces training time to 1% and improves rendering speed by up to 4,000 times for 2K resolution images in terms of FPS. Code and viewer can be found in https://github.com/Srameo/LE3D .