Unbiasing on the Fly: Explanation-Guided Human Oversight of Machine Learning System Decisions

TL;DR

This paper introduces a real-time, explanation-guided human oversight framework for detecting and correcting discriminatory decisions in deployed machine learning systems, enhancing fairness and trust.

Contribution

It proposes a novel on-the-fly monitoring framework using counterfactual explanations and human-in-the-loop intervention to mitigate discrimination during ML system deployment.

Findings

Framework effectively flags discriminatory outcomes in real-time

Human reviewers can override ML decisions to ensure fairness

Promising approach for responsible ML deployment

Abstract

The widespread adoption of ML systems across critical domains like hiring, finance, and healthcare raises growing concerns about their potential for discriminatory decision-making based on protected attributes. While efforts to ensure fairness during development are crucial, they leave deployed ML systems vulnerable to potentially exhibiting discrimination during their operations. To address this gap, we propose a novel framework for on-the-fly tracking and correction of discrimination in deployed ML systems. Leveraging counterfactual explanations, the framework continuously monitors the predictions made by an ML system and flags discriminatory outcomes. When flagged, post-hoc explanations related to the original prediction and the counterfactual alternatives are presented to a human reviewer for real-time intervention. This human-in-the-loop approach empowers reviewers to accept or override the ML system decision, enabling fair and responsible ML operation under dynamic settings. While further work is needed for validation and refinement, this framework offers a promising avenue for mitigating discrimination and building trust in ML systems deployed in a wide range of domains.