Scaling Audio-Visual Quality Assessment Dataset via Crowdsourcing

TL;DR

This paper introduces a scalable, crowdsourced approach to creating a large, diverse audio-visual quality assessment dataset, enabling better model development and multimodal perception research.

Contribution

It presents a practical crowdsourcing framework and data preparation strategy to build the largest AVQA dataset with extensive annotations.

Findings

Validated the crowdsourcing approach for reliable AVQA annotations

Created YT-NTU-AVQ, the largest diverse AVQA dataset to date

Enabled new research avenues in multimodal perception mechanisms

Abstract

Audio-visual quality assessment (AVQA) research has been stalled by limitations of existing datasets: they are typically small in scale, with insufficient diversity in content and quality, and annotated only with overall scores. These shortcomings provide limited support for model development and multimodal perception research. We propose a practical approach for AVQA dataset construction. First, we design a crowdsourced subjective experiment framework for AVQA, breaks the constraints of in-lab settings and achieves reliable annotation across varied environments. Second, a systematic data preparation strategy is further employed to ensure broad coverage of both quality levels and semantic scenarios. Third, we extend the dataset with additional annotations, enabling research on multimodal perception mechanisms and their relation to content. Finally, we validate this approach through YT-NTU-AVQ, the largest and most diverse AVQA dataset to date, consisting of 1,620 user-generated audio and video (A/V) sequences. The dataset and platform code are available at https://github.com/renyu12/YT-NTU-AVQ