Mitigating stereotypical biases in text to image generative systems

TL;DR

This paper introduces a finetuning method using synthetic diverse data to reduce social biases in text-to-image models, improving fairness across skin tones and genders.

Contribution

It proposes a novel diversity finetuning approach that significantly enhances fairness metrics in generative models by using synthetic, diverse training data.

Findings

150% improvement in skin tone fairness metric

97.7% improvement in gender fairness metric

More diverse and representative generated images

Abstract

State-of-the-art generative text-to-image models are known to exhibit social biases and over-represent certain groups like people of perceived lighter skin tones and men in their outcomes. In this work, we propose a method to mitigate such biases and ensure that the outcomes are fair across different groups of people. We do this by finetuning text-to-image models on synthetic data that varies in perceived skin tones and genders constructed from diverse text prompts. These text prompts are constructed from multiplicative combinations of ethnicities, genders, professions, age groups, and so on, resulting in diverse synthetic data. Our diversity finetuned (DFT) model improves the group fairness metric by 150% for perceived skin tone and 97.7% for perceived gender. Compared to baselines, DFT models generate more people with perceived darker skin tone and more women. To foster open research, we will release all text prompts and code to generate training images.