PhotonSplat: 3D Scene Reconstruction and Colorization from SPAD Sensors

TL;DR

PhotonSplat introduces a novel neural rendering framework that reconstructs and colorizes 3D scenes from SPAD sensor data, effectively handling noise and motion blur, and enabling dynamic scene modeling and various downstream tasks.

Contribution

The paper presents PhotonSplat, a new method for 3D scene reconstruction from SPAD binary images, including a noise reduction technique and dynamic scene extension, with a new real-world dataset.

Findings

Effective noise reduction in SPAD-based reconstructions.

Successful colorization and segmentation from blurry images.

Extension to dynamic scenes with moving objects.

Abstract

Advances in 3D reconstruction using neural rendering have enabled high-quality 3D capture. However, they often fail when the input imagery is corrupted by motion blur, due to fast motion of the camera or the objects in the scene. This work advances neural rendering techniques in such scenarios by using single-photon avalanche diode (SPAD) arrays, an emerging sensing technology capable of sensing images at extremely high speeds. However, the use of SPADs presents its own set of unique challenges in the form of binary images, that are driven by stochastic photon arrivals. To address this, we introduce PhotonSplat, a framework designed to reconstruct 3D scenes directly from SPAD binary images, effectively navigating the noise vs. blur trade-off. Our approach incorporates a novel 3D spatial filtering technique to reduce noise in the renderings. The framework also supports both no-reference using generative priors and reference-based colorization from a single blurry image, enabling downstream applications such as segmentation, object detection and appearance editing tasks. Additionally, we extend our method to incorporate dynamic scene representations, making it suitable for scenes with moving objects. We further contribute PhotonScenes, a real-world multi-view dataset captured with the SPAD sensors.