The Bias Amplification Paradox in Text-to-Image Generation

TL;DR

This paper investigates bias amplification in text-to-image models, revealing that apparent bias increases are largely due to differences in data and prompts, emphasizing the importance of accounting for distributional shifts in bias analysis.

Contribution

The study identifies how discrepancies between training captions and prompts influence bias measurements, providing a nuanced understanding of bias amplification in generative models.

Findings

Bias amplification appears significant but is reduced after accounting for text distribution differences.

Training captions often contain explicit gender info, unlike prompts, causing distribution shifts.

Addressing confounding factors is crucial for accurate bias evaluation in generative models.

Abstract

Bias amplification is a phenomenon in which models exacerbate biases or stereotypes present in the training data. In this paper, we study bias amplification in the text-to-image domain using Stable Diffusion by comparing gender ratios in training vs. generated images. We find that the model appears to amplify gender-occupation biases found in the training data (LAION) considerably. However, we discover that amplification can be largely attributed to discrepancies between training captions and model prompts. For example, an inherent difference is that captions from the training data often contain explicit gender information while our prompts do not, which leads to a distribution shift and consequently inflates bias measures. Once we account for distributional differences between texts used for training and generation when evaluating amplification, we observe that amplification decreases drastically. Our findings illustrate the challenges of comparing biases in models and their training data, and highlight confounding factors that impact analyses.