Empirical Likelihood-Based Fairness Auditing: Distribution-Free Certification and Flagging

TL;DR

This paper introduces a non-parametric, empirical likelihood-based framework for fairness auditing in machine learning, enabling distribution-free certification and flagging of biased subpopulations with improved efficiency.

Contribution

It develops a novel EL-based method that provides valid, distribution-free fairness inference and scalable subpopulation bias detection, outperforming traditional bootstrap approaches.

Findings

Successfully flagged intersectional biases in the COMPAS dataset.

Achieved more accurate coverage rates than bootstrap methods.

Reduced computational latency significantly.

Abstract

Machine learning models in high-stakes applications, such as recidivism prediction and automated personnel selection, often exhibit systematic performance disparities across sensitive subpopulations, raising critical concerns regarding algorithmic bias. Fairness auditing addresses these risks through two primary functions: certification, which verifies adherence to fairness constraints; and flagging, which isolates specific demographic groups experiencing disparate treatment. However, existing auditing techniques are frequently limited by restrictive distributional assumptions or prohibitive computational overhead. We propose a novel empirical likelihood-based (EL) framework that constructs robust statistical measures for model performance disparities. Unlike traditional methods, our approach is non-parametric; the proposed disparity statistics follow asymptotically chi-square or mixed chi-square distributions, ensuring valid inference without assuming underlying data distributions. This framework uses a constrained optimization profile that admits stable numerical solutions, facilitating both large-scale certification and efficient subpopulation discovery. Empirically, the EL methods outperform bootstrap-based approaches, yielding coverage rates closer to nominal levels while reducing computational latency by several orders of magnitude. We demonstrate the practical utility of this framework on the COMPAS dataset, where it successfully flags intersectional biases, specifically identifying a significantly higher positive prediction rate for African-American males under 25 and a systemic under-prediction for Caucasian females relative to the population mean.