MantissaCam: Learning Snapshot High-dynamic-range Imaging with Perceptually-based In-pixel Irradiance Encoding

TL;DR

MantissaCam introduces a neural network-based irradiance unwrapping algorithm and a perceptually-inspired encoding scheme to improve snapshot HDR imaging, achieving state-of-the-art results with a prototype sensor.

Contribution

The paper presents a novel neural network algorithm and a perceptual encoding scheme for HDR imaging that surpasses previous methods in accuracy and efficiency.

Findings

Outperforms previous irradiance unwrapping methods.

Achieves state-of-the-art results in snapshot HDR imaging.

Demonstrated effectiveness on simulated and real sensor data.

Abstract

The ability to image high-dynamic-range (HDR) scenes is crucial in many computer vision applications. The dynamic range of conventional sensors, however, is fundamentally limited by their well capacity, resulting in saturation of bright scene parts. To overcome this limitation, emerging sensors offer in-pixel processing capabilities to encode the incident irradiance. Among the most promising encoding schemes is modulo wrapping, which results in a computational photography problem where the HDR scene is computed by an irradiance unwrapping algorithm from the wrapped low-dynamic-range (LDR) sensor image. Here, we design a neural network--based algorithm that outperforms previous irradiance unwrapping methods and we design a perceptually inspired "mantissa" encoding scheme that more efficiently wraps an HDR scene into an LDR sensor. Combined with our reconstruction framework, MantissaCam achieves state-of-the-art results among modulo-type snapshot HDR imaging approaches. We demonstrate the efficacy of our method in simulation and show benefits of our algorithm on modulo images captured with a prototype implemented with a programmable sensor.