Retiring $\Delta$DP: New Distribution-Level Metrics for Demographic Parity

TL;DR

This paper introduces two new distribution-level metrics, ABPC and ABCC, to accurately measure demographic parity violations in machine learning, addressing limitations of the traditional $ ext{Delta} DP$ metric.

Contribution

The paper proposes two novel metrics, ABPC and ABCC, that precisely quantify demographic parity violations at the distribution level, overcoming issues with $ ext{Delta} DP$.

Findings

ABPC and ABCC provide zero violation guarantees when their values are zero.

The new metrics reveal different fairness behaviors in existing models.

They are robust to classification threshold adjustments.

Abstract

Demographic parity is the most widely recognized measure of group fairness in machine learning, which ensures equal treatment of different demographic groups. Numerous works aim to achieve demographic parity by pursuing the commonly used metric $\Delta DP$. Unfortunately, in this paper, we reveal that the fairness metric $\Delta DP$ can not precisely measure the violation of demographic parity, because it inherently has the following drawbacks: i) zero-value $\Delta DP$ does not guarantee zero violation of demographic parity, ii) $\Delta DP$ values can vary with different classification thresholds. To this end, we propose two new fairness metrics, Area Between Probability density function Curves (ABPC) and Area Between Cumulative density function Curves (ABCC), to precisely measure the violation of demographic parity at the distribution level. The new fairness metrics directly measure the difference between the distributions of the prediction probability for different demographic groups. Thus our proposed new metrics enjoy: i) zero-value ABCC/ABPC guarantees zero violation of demographic parity; ii) ABCC/ABPC guarantees demographic parity while the classification thresholds are adjusted. We further re-evaluate the existing fair models with our proposed fairness metrics and observe different fairness behaviors of those models under the new metrics. The code is available at https://github.com/ahxt/new_metric_for_demographic_parity