Transforming Video Subjective Testing with Training, Engagement, and Real-Time Feedback
Kumar Rahul, Sriram Sethuraman, Andrew Segall, Yixu Chen

TL;DR
This paper introduces an integrated framework for subjective video quality assessment that combines training, real-time engagement feedback, and efficient comparison methods to improve data reliability and reduce non-monotonic ratings.
Contribution
It presents a novel 3-phase approach with automated training, attention scoring, and streamlined comparisons, significantly enhancing the accuracy and consistency of subjective video quality measurements.
Findings
Training quiz improves data quality and accuracy.
Real-time feedback further enhances rating monotonicity.
The approach reduces non-monotonic cases in high-quality content.
Abstract
Subjective video quality assessment is crucial for optimizing streaming and compression, yet traditional protocols face limitations in capturing nuanced perceptual differences and ensuring reliable user input. We propose an integrated framework that enhances rater training, enforces attention through real-time scoring, and streamlines pairwise comparisons to recover quality scores with fewer comparisons. Participants first undergo an automated training quiz to learn key video quality indicators (e.g., compression artifacts) and verify their readiness. During the test, a real-time attention scoring mechanism, using "golden" video pairs, monitors and reinforces rater focus by applying penalties for lapses. An efficient chain-based pairwise comparison procedure is then employed, yielding quality scores in Just-Objectionable-Differences (JOD) units. Experiments comparing three groups (no…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsImage and Video Quality Assessment · Visual Attention and Saliency Detection · Video Coding and Compression Technologies
