Goldilocks: Consistent Crowdsourced Scalar Annotations with Relative Uncertainty

TL;DR

Goldilocks is a new crowd rating method that improves consistency and captures uncertainty by grounding scales with examples and using a two-step bounding process across diverse domains.

Contribution

It introduces a novel elicitation technique that distinguishes inherent ambiguity from annotator disagreement, enhancing the quality of scalar annotations.

Findings

Improves consistency in subjective rating domains.

Captures different sources of uncertainty with item ranges.

Enhances estimates of pairwise relationship distributions.

Abstract

Human ratings have become a crucial resource for training and evaluating machine learning systems. However, traditional elicitation methods for absolute and comparative rating suffer from issues with consistency and often do not distinguish between uncertainty due to disagreement between annotators and ambiguity inherent to the item being rated. In this work, we present Goldilocks, a novel crowd rating elicitation technique for collecting calibrated scalar annotations that also distinguishes inherent ambiguity from inter-annotator disagreement. We introduce two main ideas: grounding absolute rating scales with examples and using a two-step bounding process to establish a range for an item's placement. We test our designs in three domains: judging toxicity of online comments, estimating satiety of food depicted in images, and estimating age based on portraits. We show that (1) Goldilocks can improve consistency in domains where interpretation of the scale is not universal, and that (2) representing items with ranges lets us simultaneously capture different sources of uncertainty leading to better estimates of pairwise relationship distributions.