Subjective Assessment Experiments That Recruit Few Observers With Repetitions (FOWR)

TL;DR

This paper introduces a new subjective assessment method that involves few observers with multiple repetitions, enabling quick, reliable evaluation of stimuli comparable to objective metrics.

Contribution

It proposes a simple, efficient test design with few observers and repetitions that accurately characterizes subjective assessments and can be used for pre-testing new technologies.

Findings

Results are comparable to high-performing objective metrics

Few observers with multiple repetitions suffice for reliable assessment

Method simplifies and accelerates subjective testing processes

Abstract

Recent studies have shown that it is possible to characterize subject bias and variance in subjective assessment tests. Apparent differences among subjects can, for the most part, be explained by random factors. Building on that theory, we propose a subjective test design where three to four team members each rate the stimuli multiple times. The results are comparable to a high performing objective metric. This provides a quick and simple way to analyze new technologies and perform pre-tests for subjective assessment.