Perceptions of the Fairness Impacts of Multiplicity in Machine Learning

TL;DR

This study investigates how lay stakeholders perceive the fairness implications of multiplicity in machine learning models, revealing concerns about fairness but not about model use, and emphasizing the need for intentional handling of multiplicity.

Contribution

It provides empirical insights into stakeholder perceptions of multiplicity's fairness impacts and highlights preferences for addressing multiplicity in ML.

Findings

Stakeholders see multiplicity as a fairness threat.

Participants oppose ignoring multiplicity or randomization.

Perceptions vary with task stakes and uncertainty.

Abstract

Machine learning (ML) is increasingly used in high-stakes settings, yet multiplicity - the existence of multiple good models - means that some predictions are essentially arbitrary. ML researchers and philosophers posit that multiplicity poses a fairness risk, but no studies have investigated whether stakeholders agree. In this work, we conduct a survey to see how multiplicity impacts lay stakeholders' - i.e., decision subjects' - perceptions of ML fairness, and which approaches to address multiplicity they prefer. We investigate how these perceptions are modulated by task characteristics (e.g., stakes and uncertainty). Survey respondents think that multiplicity threatens the fairness of model outcomes, but not the appropriateness of using the model, even though existing work suggests the opposite. Participants are strongly against resolving multiplicity by using a single model (effectively ignoring multiplicity) or by randomizing the outcomes. Our results indicate that model developers should be intentional about dealing with multiplicity in order to maintain fairness.