Counterfactual Fairness with Graph Uncertainty

TL;DR

This paper introduces CF-GU, a method that incorporates causal graph uncertainty into counterfactual fairness evaluations, providing more reliable bias assessments in machine learning models.

Contribution

It proposes a novel approach that accounts for uncertainty in causal graph specification, enhancing the robustness of counterfactual fairness audits.

Findings

CF-GU effectively captures graph uncertainty in bias evaluation.

Experiments demonstrate high-confidence bias detection on real datasets.

The method supports minimal domain knowledge for reliable audits.

Abstract

Evaluating machine learning (ML) model bias is key to building trustworthy and robust ML systems. Counterfactual Fairness (CF) audits allow the measurement of bias of ML models with a causal framework, yet their conclusions rely on a single causal graph that is rarely known with certainty in real-world scenarios. We propose CF with Graph Uncertainty (CF-GU), a bias evaluation procedure that incorporates the uncertainty of specifying a causal graph into CF. CF-GU (i) bootstraps a Causal Discovery algorithm under domain knowledge constraints to produce a bag of plausible Directed Acyclic Graphs (DAGs), (ii) quantifies graph uncertainty with the normalized Shannon entropy, and (iii) provides confidence bounds on CF metrics. Experiments on synthetic data show how contrasting domain knowledge assumptions support or refute audits of CF, while experiments on real-world data (COMPAS and Adult datasets) pinpoint well-known biases with high confidence, even when supplied with minimal domain knowledge constraints.