Image Denoising and the Generative Accumulation of Photons

TL;DR

This paper introduces GAP, a novel self-supervised generative model for photon accumulation in images, improving denoising and sampling by modeling image formation as sequential photon addition.

Contribution

It proposes a new perspective on image denoising via photon accumulation, introduces a self-supervised training strategy, and develops a generative sampling method for noisy images.

Findings

Outperforms supervised, self-supervised, and unsupervised baselines.

Provides a new generative model for photon-based image formation.

Demonstrates effectiveness on fluorescence microscopy datasets.

Abstract

We present a fresh perspective on shot noise corrupted images and noise removal. By viewing image formation as the sequential accumulation of photons on a detector grid, we show that a network trained to predict where the next photon could arrive is in fact solving the minimum mean square error (MMSE) denoising task. This new perspective allows us to make three contributions: We present a new strategy for self-supervised denoising, We present a new method for sampling from the posterior of possible solutions by iteratively sampling and adding small numbers of photons to the image. We derive a full generative model by starting this process from an empty canvas. We call this approach generative accumulation of photons (GAP). We evaluate our method quantitatively and qualitatively on 4 new fluorescence microscopy datasets, which will be made available to the community. We find that it outperforms supervised, self-supervised and unsupervised baselines or performs on-par.