Granular feedback merits sophisticated aggregation

TL;DR

This paper demonstrates that for granular feedback, advanced aggregation methods significantly outperform simple regularized averaging, especially with higher feedback granularity, leading to more accurate population distribution estimates with fewer individuals.

Contribution

It introduces and empirically validates sophisticated aggregation techniques that outperform regularized averaging for granular feedback, particularly with multi-point feedback.

Findings

Sophisticated methods outperform regularized averaging with five-point feedback.

With binary feedback, sophistication offers minimal improvement.

Higher feedback granularity enhances the benefits of advanced aggregation techniques.

Abstract

Human feedback is increasingly used across diverse applications like training AI models, developing recommender systems, and measuring public opinion -- with granular feedback often being preferred over binary feedback for its greater informativeness. While it is easy to accurately estimate a population's distribution of feedback given feedback from a large number of individuals, cost constraints typically necessitate using smaller groups. A simple method to approximate the population distribution is regularized averaging: compute the empirical distribution and regularize it toward a prior. Can we do better? As we will discuss, the answer to this question depends on feedback granularity. Suppose one wants to predict a population's distribution of feedback using feedback from a limited number of individuals. We show that, as feedback granularity increases, one can substantially improve upon predictions of regularized averaging by combining individuals' feedback in ways more sophisticated than regularized averaging. Our empirical analysis using questions on social attitudes confirms this pattern. In particular, with binary feedback, sophistication barely reduces the number of individuals required to attain a fixed level of performance. By contrast, with five-point feedback, sophisticated methods match the performance of regularized averaging with about half as many individuals.