OpenDenoising: an Extensible Benchmark for Building Comparative Studies of Image Denoisers

TL;DR

OpenDenoising introduces an extensible benchmark for evaluating and comparing image denoisers, emphasizing real-world noise performance and providing a reproducible framework for future studies.

Contribution

The paper presents a new benchmarking tool for image denoisers that facilitates reproducibility, extension, and evaluation on real-world noises, highlighting the importance of realistic testing.

Findings

MWCNN outperforms others on real-world noise

MWCNN is second most computationally efficient

Benchmarking reveals low correlation across different noise types

Abstract

Image denoising has recently taken a leap forward due to machine learning. However, image denoisers, both expert-based and learning-based, are mostly tested on well-behaved generated noises (usually Gaussian) rather than on real-life noises, making performance comparisons difficult in real-world conditions. This is especially true for learning-based denoisers which performance depends on training data. Hence, choosing which method to use for a specific denoising problem is difficult. This paper proposes a comparative study of existing denoisers, as well as an extensible open tool that makes it possible to reproduce and extend the study. MWCNN is shown to outperform other methods when trained for a real-world image interception noise, and additionally is the second least compute hungry of the tested methods. To evaluate the robustness of conclusions, three test sets are compared. A Kendall's Tau correlation of only 60% is obtained on methods ranking between noise types, demonstrating the need for a benchmarking tool.