Panoramas from Photons

TL;DR

This paper introduces a novel method for scene reconstruction and panorama creation using single-photon camera data, effectively handling high-speed motion and low-light conditions by iteratively aggregating photon frames.

Contribution

The paper presents a new approach for estimating scene motion and creating panoramas from single-photon camera data under challenging conditions, addressing noise and motion issues.

Findings

Successfully creates high-quality panoramas under fast motion and low light

Demonstrates super-resolution results with a custom single-photon camera

Provides a method that iteratively improves motion estimates from photon data

Abstract

Scene reconstruction in the presence of high-speed motion and low illumination is important in many applications such as augmented and virtual reality, drone navigation, and autonomous robotics. Traditional motion estimation techniques fail in such conditions, suffering from too much blur in the presence of high-speed motion and strong noise in low-light conditions. Single-photon cameras have recently emerged as a promising technology capable of capturing hundreds of thousands of photon frames per second thanks to their high speed and extreme sensitivity. Unfortunately, traditional computer vision techniques are not well suited for dealing with the binary-valued photon data captured by these cameras because these are corrupted by extreme Poisson noise. Here we present a method capable of estimating extreme scene motion under challenging conditions, such as low light or high dynamic range, from a sequence of high-speed image frames such as those captured by a single-photon camera. Our method relies on iteratively improving a motion estimate by grouping and aggregating frames after-the-fact, in a stratified manner. We demonstrate the creation of high-quality panoramas under fast motion and extremely low light, and super-resolution results using a custom single-photon camera prototype. For code and supplemental material see our $\href{https://wisionlab.com/project/panoramas-from-photons/}{\text{project webpage}}$.