Testing for Causal Fairness

TL;DR

This paper introduces a distribution-based approach to causal fairness testing that improves reliability in high-dimensional data by assessing the closeness of factual and counterfactual distributions using a new statistical test.

Contribution

It proposes a novel distributional closeness testing framework with a new statistic, N-TE, and establishes its theoretical consistency for more trustworthy fairness analysis.

Findings

CF-CLOT effectively detects unfair attributes in real-world data.

The N-TE statistic is theoretically consistent for fairness testing.

The method is flexible and sensitive to fairness violations.

Abstract

Causality is widely used in fairness analysis to prevent discrimination on sensitive attributes, such as genders in career recruitment and races in crime prediction. However, the current data-based Potential Outcomes Framework (POF) often leads to untrustworthy fairness analysis results when handling high-dimensional data. To address this, we introduce a distribution-based POF that transform fairness analysis into Distributional Closeness Testing (DCT) by intervening on sensitive attributes. We define counterfactual closeness fairness as the null hypothesis of DCT, where a sensitive attribute is considered fair if its factual and counterfactual potential outcome distributions are sufficiently close. We introduce the Norm-Adaptive Maximum Mean Discrepancy Treatment Effect (N-TE) as a statistic for measuring distributional closeness and apply DCT using the empirical estimator of NTE, referred to Counterfactual Fairness-CLOseness Testing ($\textrm{CF-CLOT}$). To ensure the trustworthiness of testing results, we establish the testing consistency of N-TE through rigorous theoretical analysis. $\textrm{CF-CLOT}$ demonstrates sensitivity in fairness analysis through the flexibility of the closeness parameter $\epsilon$. Unfair sensitive attributes have been successfully tested by $\textrm{CF-CLOT}$ in extensive experiments across various real-world scenarios, which validate the consistency of the testing.