Massive Online Crowdsourced Study of Subjective and Objective Picture Quality

TL;DR

This paper introduces a large-scale, real-world image quality database and crowdsourced subjective study, revealing insights into complex distortions in images captured by mobile devices and evaluating blind quality assessment algorithms.

Contribution

The paper presents a new diverse database of authentic images with crowdsourced opinion scores and analyzes the performance of quality prediction models on real-world distortions.

Findings

High internal consistency of crowdsourced subjective scores

Complex distortions challenge existing quality assessment algorithms

Large-scale database enables better understanding of real-world image quality

Abstract

Most publicly available image quality databases have been created under highly controlled conditions by introducing graded simulated distortions onto high-quality photographs. However, images captured using typical real-world mobile camera devices are usually afflicted by complex mixtures of multiple distortions, which are not necessarily well-modeled by the synthetic distortions found in existing databases. The originators of existing legacy databases usually conducted human psychometric studies to obtain statistically meaningful sets of human opinion scores on images in a stringently controlled visual environment, resulting in small data collections relative to other kinds of image analysis databases. Towards overcoming these limitations, we designed and created a new database that we call the LIVE In the Wild Image Quality Challenge Database, which contains widely diverse authentic image distortions on a large number of images captured using a representative variety of modern mobile devices. We also designed and implemented a new online crowdsourcing system, which we have used to conduct a very large-scale, multi-month image quality assessment subjective study. Our database consists of over 350000 opinion scores on 1162 images evaluated by over 7000 unique human observers. Despite the lack of control over the experimental environments of the numerous study participants, we demonstrate excellent internal consistency of the subjective dataset. We also evaluate several top-performing blind Image Quality Assessment algorithms on it and present insights on how mixtures of distortions challenge both end users as well as automatic perceptual quality prediction models.