Residual Unfairness in Fair Machine Learning from Prejudiced Data

TL;DR

This paper investigates how systematic data censoring due to biased policies can cause residual unfairness in fairness-adjusted classifiers, often perpetuating existing injustices even after applying fairness corrections.

Contribution

It provides a theoretical analysis of residual unfairness caused by data censoring, criteria for its occurrence, and methods to adjust fairness metrics considering censoring effects.

Findings

Fairness adjustments can perpetuate existing biases due to data censoring.

Residual unfairness may persist even after fairness correction under certain conditions.

Sample reweighting can help estimate and mitigate fairness issues caused by censoring.

Abstract

Recent work in fairness in machine learning has proposed adjusting for fairness by equalizing accuracy metrics across groups and has also studied how datasets affected by historical prejudices may lead to unfair decision policies. We connect these lines of work and study the residual unfairness that arises when a fairness-adjusted predictor is not actually fair on the target population due to systematic censoring of training data by existing biased policies. This scenario is particularly common in the same applications where fairness is a concern. We characterize theoretically the impact of such censoring on standard fairness metrics for binary classifiers and provide criteria for when residual unfairness may or may not appear. We prove that, under certain conditions, fairness-adjusted classifiers will in fact induce residual unfairness that perpetuates the same injustices, against the same groups, that biased the data to begin with, thus showing that even state-of-the-art fair machine learning can have a "bias in, bias out" property. When certain benchmark data is available, we show how sample reweighting can estimate and adjust fairness metrics while accounting for censoring. We use this to study the case of Stop, Question, and Frisk (SQF) and demonstrate that attempting to adjust for fairness perpetuates the same injustices that the policy is infamous for.