Benchmarking Denoising Algorithms with Real Photographs

TL;DR

This paper introduces a new benchmark dataset and methodology for evaluating image denoising algorithms on real photographs, addressing the limitations of synthetic noise evaluations.

Contribution

It develops a practical approach for benchmarking denoising methods on real images and introduces the Darmstadt Noise Dataset (DND) for this purpose.

Findings

Recent denoising techniques underperform compared to BM3D on real noisy photographs.

The proposed methodology effectively derives ground truth from real images.

Benchmark results highlight the gap between synthetic noise performance and real-world effectiveness.

Abstract

Lacking realistic ground truth data, image denoising techniques are traditionally evaluated on images corrupted by synthesized i.i.d. Gaussian noise. We aim to obviate this unrealistic setting by developing a methodology for benchmarking denoising techniques on real photographs. We capture pairs of images with different ISO values and appropriately adjusted exposure times, where the nearly noise-free low-ISO image serves as reference. To derive the ground truth, careful post-processing is needed. We correct spatial misalignment, cope with inaccuracies in the exposure parameters through a linear intensity transform based on a novel heteroscedastic Tobit regression model, and remove residual low-frequency bias that stems, e.g., from minor illumination changes. We then capture a novel benchmark dataset, the Darmstadt Noise Dataset (DND), with consumer cameras of differing sensor sizes. One interesting finding is that various recent techniques that perform well on synthetic noise are clearly outperformed by BM3D on photographs with real noise. Our benchmark delineates realistic evaluation scenarios that deviate strongly from those commonly used in the scientific literature.