Uncertainty-driven Sampling for Efficient Pairwise Comparison Subjective Assessment

TL;DR

This paper introduces an uncertainty-driven sampling method that leverages deep learning to efficiently select image pairs for subjective quality assessment, significantly reducing the number of comparisons needed while maintaining high accuracy.

Contribution

It presents a novel uncertainty-based sampling approach that models human preference uncertainty to optimize pairwise comparison assessments in image quality evaluation.

Findings

Reduces the number of comparisons needed for accurate assessments

Outperforms traditional active sampling methods in efficiency

Maintains high accuracy in subjective image quality evaluation

Abstract

Assessing image quality is crucial in image processing tasks such as compression, super-resolution, and denoising. While subjective assessments involving human evaluators provide the most accurate quality scores, they are impractical for large-scale or continuous evaluations due to their high cost and time requirements. Pairwise comparison subjective assessment tests, which rank image pairs instead of assigning scores, offer more reliability and accuracy but require numerous comparisons, leading to high costs. Although objective quality metrics are more efficient, they lack the precision of subjective tests, which are essential for benchmarking and training learning-based quality metrics. This paper proposes an uncertainty-based sampling method to optimize the pairwise comparison subjective assessment process. By utilizing deep learning models to estimate human preferences and identify pairs that need human labeling, the approach reduces the number of required comparisons while maintaining high accuracy. The key contributions include modeling uncertainty for accurate preference predictions and for pairwise sampling. The experimental results demonstrate superior performance of the proposed approach compared to traditional active sampling methods. Software is publicly available at: shimamohammadi/LBPS-EIC