Jury Learning: Integrating Dissenting Voices into Machine Learning Models

TL;DR

Jury learning is a new supervised machine learning approach that explicitly incorporates diverse societal perspectives into model predictions by simulating a jury of annotators, allowing for dynamic and interpretable decision-making.

Contribution

The paper introduces a deep learning architecture for jury learning that models individual annotators, enabling explicit representation of dissent and societal diversity in ML predictions.

Findings

Enables explicit modeling of minority and dissenting opinions.

Allows dynamic adaptation of jury composition.

Provides tools for counterfactual analysis and visualization.

Abstract

Whose labels should a machine learning (ML) algorithm learn to emulate? For ML tasks ranging from online comment toxicity to misinformation detection to medical diagnosis, different groups in society may have irreconcilable disagreements about ground truth labels. Supervised ML today resolves these label disagreements implicitly using majority vote, which overrides minority groups' labels. We introduce jury learning, a supervised ML approach that resolves these disagreements explicitly through the metaphor of a jury: defining which people or groups, in what proportion, determine the classifier's prediction. For example, a jury learning model for online toxicity might centrally feature women and Black jurors, who are commonly targets of online harassment. To enable jury learning, we contribute a deep learning architecture that models every annotator in a dataset, samples from annotators' models to populate the jury, then runs inference to classify. Our architecture enables juries that dynamically adapt their composition, explore counterfactuals, and visualize dissent.