Judgment Sieve: Reducing Uncertainty in Group Judgments through Interventions Targeting Ambiguity versus Disagreement

TL;DR

This paper introduces Judgment Sieve, a targeted workflow that reduces group judgment uncertainty by identifying and addressing specific sources of ambiguity or disagreement, improving decision quality in tasks like similarity rating and toxicity detection.

Contribution

It presents a novel method to measure and target different sources of uncertainty, enabling more effective interventions tailored to each case in group judgment tasks.

Findings

Targeted interventions significantly reduced uncertainty in top cases.

Ambiguity reduction of 21.4% and 25.7% in two tasks.

Disagreement reduction of 22.2% and 11.2% in two tasks.

Abstract

When groups of people are tasked with making a judgment, the issue of uncertainty often arises. Existing methods to reduce uncertainty typically focus on iteratively improving specificity in the overall task instruction. However, uncertainty can arise from multiple sources, such as ambiguity of the item being judged due to limited context, or disagreements among the participants due to different perspectives and an under-specified task. A one-size-fits-all intervention may be ineffective if it is not targeted to the right source of uncertainty. In this paper we introduce a new workflow, Judgment Sieve, to reduce uncertainty in tasks involving group judgment in a targeted manner. By utilizing measurements that separate different sources of uncertainty during an initial round of judgment elicitation, we can then select a targeted intervention adding context or deliberation to most effectively reduce uncertainty on each item being judged. We test our approach on two tasks: rating word pair similarity and toxicity of online comments, showing that targeted interventions reduced uncertainty for the most uncertain cases. In the top 10% of cases, we saw an ambiguity reduction of 21.4% and 25.7%, and a disagreement reduction of 22.2% and 11.2% for the two tasks respectively. We also found through a simulation that our targeted approach reduced the average uncertainty scores for both sources of uncertainty as opposed to uniform approaches where reductions in average uncertainty from one source came with an increase for the other.