Standardized Interpretable Fairness Measures for Continuous Risk Scores

TL;DR

This paper introduces standardized fairness measures for continuous risk scores using Wasserstein distance, enabling better interpretation, comparison, and detection of biases across models and datasets.

Contribution

It proposes a new standardized fairness measure based on Wasserstein distance, improving interpretability and bias detection over existing ROC-based measures.

Findings

Standardized measures are easily computable.

They outperform ROC-based fairness measures.

They effectively quantify and compare biases.

Abstract

We propose a standardized version of fairness measures for continuous scores with a reasonable interpretation based on the Wasserstein distance. Our measures are easily computable and well suited for quantifying and interpreting the strength of group disparities as well as for comparing biases across different models, datasets, or time points. We derive a link between the different families of existing fairness measures for scores and show that the proposed standardized fairness measures outperform ROC-based fairness measures because they are more explicit and can quantify significant biases that ROC-based fairness measures miss.