In-place Double Stimulus Methodology for Subjective Assessment of High Quality Images

TL;DR

This paper presents a new in-place double stimulus methodology for subjective image quality assessment, improving detection of subtle differences in high-quality images and providing a comprehensive dataset for evaluation.

Contribution

The paper introduces the IDSQS methodology, a novel in-place double stimulus protocol, and provides a large-scale crowdsourced dataset with a Beta distribution model for quality scores.

Findings

IDSQS achieves high correlation with benchmark evaluations.

The dataset enables detailed analysis of perceived image quality.

Modeling with Beta distribution captures variability and consistency.

Abstract

This paper introduces a novel double stimulus subjective assessment methodology for the evaluation of high quality images to address the limitations of existing protocols in detecting subtle perceptual differences. The In-place Double Stimulus Quality Scale (IDSQS) allows subjects to alternately view a reference and a distorted image at the same spatial location, facilitating a more intuitive detection of differences in quality, especially at high to visually lossless quality levels. A large-scale crowdsourcing study employing this methodology was conducted, generating a comprehensive public dataset to evaluate perceived image quality across several compression algorithms and distortion levels. An additional contribution is the modeling of quality scores using a Beta distribution, allowing for the assessment of variability and subject consistency. Our findings demonstrate the effectiveness of the IDSQS methodology in achieving high correlation with more precise subjective evaluation benchmarks. The dataset, subjective data, and graphical user interface developed for this study are publicly available at https://github.com/shimamohammadi/IDSQS