Fairness Feedback Loops: Training on Synthetic Data Amplifies Bias

TL;DR

This paper investigates how model-induced distribution shifts can reinforce biases and unfairness over generations, and proposes a framework for tracking, understanding, and intervening to mitigate these feedback loops in machine learning systems.

Contribution

It introduces a framework to track multiple model-induced distribution shifts over generations and proposes algorithmic reparation to mitigate bias and unfairness in data ecosystems.

Findings

Model feedback loops can reduce fairness and representation.

Interventions via curated training batches can improve fairness.

Tracking MIDS helps understand bias amplification over generations.

Abstract

Model-induced distribution shifts (MIDS) occur as previous model outputs pollute new model training sets over generations of models. This is known as model collapse in the case of generative models, and performative prediction or unfairness feedback loops for supervised models. When a model induces a distribution shift, it also encodes its mistakes, biases, and unfairnesses into the ground truth of its data ecosystem. We introduce a framework that allows us to track multiple MIDS over many generations, finding that they can lead to loss in performance, fairness, and minoritized group representation, even in initially unbiased datasets. Despite these negative consequences, we identify how models might be used for positive, intentional, interventions in their data ecosystems, providing redress for historical discrimination through a framework called algorithmic reparation (AR). We simulate AR interventions by curating representative training batches for stochastic gradient descent to demonstrate how AR can improve upon the unfairnesses of models and data ecosystems subject to other MIDS. Our work takes an important step towards identifying, mitigating, and taking accountability for the unfair feedback loops enabled by the idea that ML systems are inherently neutral and objective.