A Human-in-the-Loop Fairness-Aware Model Selection Framework for Complex Fairness Objective Landscapes

TL;DR

This paper presents ManyFairHPO, a human-in-the-loop framework that treats fairness as a many-objective problem, helping practitioners navigate conflicting fairness metrics for more socially responsible model selection.

Contribution

It introduces ManyFairHPO, a novel framework that enables balancing multiple fairness objectives interactively, addressing the limitations of single-metric optimization in fairness-aware machine learning.

Findings

Effective in balancing multiple fairness metrics.

Reduces risks like self-fulfilling prophecies.

Provides interpretable insights for stakeholders.

Abstract

Fairness-aware Machine Learning (FairML) applications are often characterized by complex social objectives and legal requirements, frequently involving multiple, potentially conflicting notions of fairness. Despite the well-known Impossibility Theorem of Fairness and extensive theoretical research on the statistical and socio-technical trade-offs between fairness metrics, many FairML tools still optimize or constrain for a single fairness objective. However, this one-sided optimization can inadvertently lead to violations of other relevant notions of fairness. In this socio-technical and empirical study, we frame fairness as a many-objective (MaO) problem by treating fairness metrics as conflicting objectives. We introduce ManyFairHPO, a human-in-the-loop, fairness-aware model selection framework that enables practitioners to effectively navigate complex and nuanced fairness objective landscapes. ManyFairHPO aids in the identification, evaluation, and balancing of fairness metric conflicts and their related social consequences, leading to more informed and socially responsible model-selection decisions. Through a comprehensive empirical evaluation and a case study on the Law School Admissions problem, we demonstrate the effectiveness of ManyFairHPO in balancing multiple fairness objectives, mitigating risks such as self-fulfilling prophecies, and providing interpretable insights to guide stakeholders in making fairness-aware modeling decisions.