Failing on Bias Mitigation: A Case Study on the Challenges of Fairness in Government Data

TL;DR

This study investigates why bias mitigation techniques often fail in government data, revealing that inherent data properties like historical bias and distribution shifts are primary obstacles to achieving fairness.

Contribution

The paper provides a detailed case study showing the limitations of bias mitigation methods on government data and identifies data properties as key sources of persistent bias.

Findings

Bias mitigation often fails due to data's inherent properties.

Historical bias and data shifts are major challenges.

Mitigation efforts cannot fully overcome embedded unfairness.

Abstract

The potential for bias and unfairness in AI-supporting government services raises ethical and legal concerns. Using crime rate prediction with the Bristol City Council data as a case study, we examine how these issues persist. Rather than auditing real-world deployed systems, our goal is to understand why widely adopted bias mitigation techniques often fail when applied to government data. Our findings reveal that bias mitigation approaches applied to government data are not always effective -- not because of flaws in model architecture or metric selection, but due to the inherent properties of the data itself. Through comparing a set of comprehensive models and fairness methods, our experiments consistently show that the mitigation efforts cannot overcome the embedded unfairness in the data -- further reinforcing that the origin of bias lies in the structure and history of government datasets. We then explore the reasons for the mitigation failures in predictive models on government data and highlight the potential sources of unfairness posed by data distribution shifts, the accumulation of historical bias, and delays in data release. We also discover the limitations of the blind spots in fairness analysis and bias mitigation methods when only targeting a single sensitive feature through a set of intersectional fairness experiments. Although this study is limited to one city, the findings are highly suggestive, which can contribute to an early warning that biases in government data may persist even with standard mitigation methods.