The Effect of Enforcing Fairness on Reshaping Explanations in Machine Learning Models

TL;DR

This paper investigates how applying fairness constraints to machine learning models in healthcare impacts the stability and interpretation of feature importance explanations, emphasizing the need for joint evaluation of accuracy, fairness, and explainability.

Contribution

It provides a systematic analysis of how fairness interventions reshape Shapley-based explanations across multiple datasets and model classes.

Findings

Fairness constraints can significantly change feature importance rankings.

Changes in explanations vary across racial subgroups.

Joint consideration of accuracy, fairness, and explainability is essential.

Abstract

Trustworthy machine learning in healthcare requires strong predictive performance, fairness, and explanations. While it is known that improving fairness can affect predictive performance, little is known about how fairness improvements influence explainability, an essential ingredient for clinical trust. Clinicians may hesitate to rely on a model whose explanations shift after fairness constraints are applied. In this study, we examine how enhancing fairness through bias mitigation techniques reshapes Shapley-based feature rankings. We quantify changes in feature importance rankings after applying fairness constraints across three datasets: pediatric urinary tract infection risk, direct anticoagulant bleeding risk, and recidivism risk. We also evaluate multiple model classes on the stability of Shapley-based rankings. We find that increasing model fairness across racial subgroups can significantly alter feature importance rankings, sometimes in different ways across groups. These results highlight the need to jointly consider accuracy, fairness, and explainability in model assessment rather than in isolation.