Lucky High Dynamic Range Smartphone Imaging

TL;DR

This paper introduces a lightweight, robust HDR imaging method for smartphones that operates on raw pixels, generalizes well to real images, and enhances existing techniques without hallucination artifacts.

Contribution

It presents a novel approach using convex combinations of neighborhood pixels in bracketed exposures, suitable for mobile devices, and demonstrates zero-shot generalization to real-world smartphone images.

Findings

Achieves 3-9 stops of dynamic range extension in smartphone images.

Generalizes from synthetic training to real camera captures without fine-tuning.

Improves state-of-the-art HDR methods when trained with the proposed scheme.

Abstract

While the human eye can perceive an impressive twenty stops of dynamic range, smartphone camera sensors remain limited to about twelve stops despite decades of research. A variety of high dynamic range (HDR) image capture and processing techniques have been proposed, and, in practice, they can extend the dynamic range by 3-5 stops for handheld photography. This paper proposes an approach that robustly captures dynamic range using a handheld smartphone camera and lightweight networks suitable for running on mobile devices. Our method operates indirectly on linear raw pixels in bracketed exposures. Every pixel in the final HDR image is a convex combination of input pixels in the neighborhood, adjusted for exposure, and thus avoids hallucination artifacts typical of recent deep image synthesis networks. We validate our system on both synthetic imagery and unseen real bracketed images -- we confirm zero-shot generalization of the method to smartphone camera captures. Our iterative inference architecture is capable of processing an arbitrary number of bracketed input photos, and we show examples from capture stacks containing 3--9 images. Our training process relies only on synthetic captures yet generalizes to unseen real photos from several cameras. Moreover, we show that this training scheme improves other SOTA methods over their pretrained counterparts.