Capturing Variation and Uncertainty in Human Judgment

TL;DR

This paper demonstrates that generalized random utility models better capture the variation and uncertainty in human judgment rankings across different domains compared to classical models.

Contribution

It introduces the effectiveness of flexible ranking models in representing human preferences and highlights limitations of traditional statistical ranking models.

Findings

Generalized random utility models reveal distinctive patterns in human judgment.

Classical models fail to capture key features of human-generated rankings.

Flexible models better represent collective human preferences.

Abstract

The well-studied problem of statistical rank aggregation has been applied to comparing sports teams, information retrieval, and most recently to data generated by human judgment. Such human-generated rankings may be substantially different from traditional statistical ranking data. In this work, we show that a recently proposed generalized random utility model reveals distinctive patterns in human judgment across three different domains, and provides a succinct representation of variance in both population preferences and imperfect perception. In contrast, we also show that classical statistical ranking models fail to capture important features from human-generated input. Our work motivates the use of more flexible ranking models for representing and describing the collective preferences or decision-making of human participants.