Equal Affection or Random Selection: the Quality of Subjective Feedback from a Group Perspective

TL;DR

This paper introduces a new evaluation method for subjective group feedback that distinguishes between informative and uninformative responses by analyzing respondents' predictions about others' choices, improving feedback quality assessment.

Contribution

It proposes a novel metric called f-variety that effectively differentiates informative from uninformative feedback in group surveys, even with uniform statistics.

Findings

f-variety successfully separates informative and uninformative feedback

The metric decreases as the ratio of uninformative responses increases

Case studies demonstrate the method's practical effectiveness

Abstract

In the setting where a group of agents is asked a single subjective multi-choice question (e.g. which one do you prefer? cat or dog?), we are interested in evaluating the quality of the collected feedback. However, the collected statistics are not sufficient to reflect how informative the feedback is since fully informative feedback (equal affection of the choices) and fully uninformative feedback (random selection) have the same uniform statistics. Here we distinguish the above two scenarios by additionally asking for respondents' predictions about others' choices. We assume that informative respondents' predictions strongly depend on their own choices while uninformative respondents' do not. With this assumption, we propose a new definition for uninformative feedback and correspondingly design a family of evaluation metrics, called f-variety, for group-level feedback which can 1) distinguish informative feedback and uninformative feedback (separation) even if their statistics are both uniform and 2) decrease as the ratio of uninformative respondents increases (monotonicity). We validate our approach both theoretically and numerically. Moreover, we conduct two real-world case studies about 1) comparisons about athletes and 2) comparisons about stand-up comedians to show the superiority of our approach.