FairLens: Auditing Black-box Clinical Decision Support Systems

TL;DR

FairLens is a novel methodology that enables healthcare domain experts to audit, discover, and explain biases in black-box clinical decision support systems using subgroup analysis and explainability techniques.

Contribution

It introduces a systematic approach combining data stratification and XAI to identify and interpret biases in black-box medical AI models.

Findings

Effective bias detection in clinical models

Ability to explain subgroup-specific errors

Supports trust and fairness assessment in healthcare AI

Abstract

The pervasive application of algorithmic decision-making is raising concerns on the risk of unintended bias in AI systems deployed in critical settings such as healthcare. The detection and mitigation of biased models is a very delicate task which should be tackled with care and involving domain experts in the loop. In this paper we introduce FairLens, a methodology for discovering and explaining biases. We show how our tool can be used to audit a fictional commercial black-box model acting as a clinical decision support system. In this scenario, the healthcare facility experts can use FairLens on their own historical data to discover the model's biases before incorporating it into the clinical decision flow. FairLens first stratifies the available patient data according to attributes such as age, ethnicity, gender and insurance; it then assesses the model performance on such subgroups of patients identifying those in need of expert evaluation. Finally, building on recent state-of-the-art XAI (eXplainable Artificial Intelligence) techniques, FairLens explains which elements in patients' clinical history drive the model error in the selected subgroup. Therefore, FairLens allows experts to investigate whether to trust the model and to spotlight group-specific biases that might constitute potential fairness issues.