Debiased Subjective Assessment of Real-World Image Enhancement

TL;DR

This paper introduces a debiased subjective assessment method for real-world image enhancement that automatically samples diverse images to evaluate enhancement algorithms more reliably, reducing biases present in traditional subjective testing.

Contribution

It proposes an automatic sampling approach that minimizes biases in subjective quality assessment of image enhancement, enabling more objective and reliable evaluations.

Findings

Debiased sampling improves assessment reliability.

Method applied successfully to dehazing, super-resolution, low-light enhancement.

Enhanced evaluation reduces subjective and sampling biases.

Abstract

In real-world image enhancement, it is often challenging (if not impossible) to acquire ground-truth data, preventing the adoption of distance metrics for objective quality assessment. As a result, one often resorts to subjective quality assessment, the most straightforward and reliable means of evaluating image enhancement. Conventional subjective testing requires manually pre-selecting a small set of visual examples, which may suffer from three sources of biases: 1) sampling bias due to the extremely sparse distribution of the selected samples in the image space; 2) algorithmic bias due to potential overfitting the selected samples; 3) subjective bias due to further potential cherry-picking test results. This eventually makes the field of real-world image enhancement more of an art than a science. Here we take steps towards debiasing conventional subjective assessment by automatically sampling a set of adaptive and diverse images for subsequent testing. This is achieved by casting sample selection into a joint maximization of the discrepancy between the enhancers and the diversity among the selected input images. Careful visual inspection on the resulting enhanced images provides a debiased ranking of the enhancement algorithms. We demonstrate our subjective assessment method using three popular and practically demanding image enhancement tasks: dehazing, super-resolution, and low-light enhancement.