Radiance Fields from Photons

TL;DR

This paper introduces quanta radiance fields, a new neural radiance field approach trained on individual photons from single-photon cameras, enabling high-fidelity view synthesis in challenging lighting and motion conditions.

Contribution

It develops theory and techniques for building radiance fields from photon-level data and estimates camera poses from stochastic, high-speed binary sequences.

Findings

High-quality reconstructions in low light and high dynamic range conditions.

Effective handling of high-speed motion with photon-based radiance fields.

Successful implementation with both simulations and hardware prototype.

Abstract

Neural radiance fields, or NeRFs, have become the de facto approach for high-quality view synthesis from a collection of images captured from multiple viewpoints. However, many issues remain when capturing images in-the-wild under challenging conditions, such as low light, high dynamic range, or rapid motion leading to smeared reconstructions with noticeable artifacts. In this work, we introduce quanta radiance fields, a novel class of neural radiance fields that are trained at the granularity of individual photons using single-photon cameras (SPCs). We develop theory and practical computational techniques for building radiance fields and estimating dense camera poses from unconventional, stochastic, and high-speed binary frame sequences captured by SPCs. We demonstrate, both via simulations and a SPC hardware prototype, high-fidelity reconstructions under high-speed motion, in low light, and for extreme dynamic range settings.