SPC to 3D: Novel View Synthesis from Binary SPC via I2I translation

TL;DR

This paper introduces a two-stage framework that converts binary SPC images into high-quality colorized 3D views by combining image translation with 3D scene reconstruction techniques, enabling effective novel view synthesis from photon-limited data.

Contribution

It presents a novel modular pipeline that transforms binary SPC data into colorized 3D representations, bridging the gap between photon detection and 3D scene understanding.

Findings

Significant improvement in perceptual quality of generated views.

Enhanced geometric consistency over baseline methods.

Effective colorization of binary SPC images for 3D synthesis.

Abstract

Single Photon Avalanche Diodes (SPADs) represent a cutting-edge imaging technology, capable of detecting individual photons with remarkable timing precision. Building on this sensitivity, Single Photon Cameras (SPCs) enable image capture at exceptionally high speeds under both low and high illumination. Enabling 3D reconstruction and radiance field recovery from such SPC data holds significant promise. However, the binary nature of SPC images leads to severe information loss, particularly in texture and color, making traditional 3D synthesis techniques ineffective. To address this challenge, we propose a modular two-stage framework that converts binary SPC images into high-quality colorized novel views. The first stage performs image-to-image (I2I) translation using generative models such as Pix2PixHD, converting binary SPC inputs into plausible RGB representations. The second stage employs 3D scene reconstruction techniques like Neural Radiance Fields (NeRF) or Gaussian Splatting (3DGS) to generate novel views. We validate our two-stage pipeline (Pix2PixHD + Nerf/3DGS) through extensive qualitative and quantitative experiments, demonstrating significant improvements in perceptual quality and geometric consistency over the alternative baseline.