Fair Diffusion: Instructing Text-to-Image Generation Models on Fairness

TL;DR

Fair Diffusion is a novel method that allows post-deployment bias adjustment in text-to-image models through human instructions, enabling control over fairness without retraining or data filtering.

Contribution

It introduces a new strategy for bias mitigation in generative models that operates after deployment, using human instructions to steer fairness proportions.

Findings

Enables bias shifting in generated images based on human instructions

Operates without additional data filtering or retraining

Effectively controls fairness in text-to-image generation

Abstract

Generative AI models have recently achieved astonishing results in quality and are consequently employed in a fast-growing number of applications. However, since they are highly data-driven, relying on billion-sized datasets randomly scraped from the internet, they also suffer from degenerated and biased human behavior, as we demonstrate. In fact, they may even reinforce such biases. To not only uncover but also combat these undesired effects, we present a novel strategy, called Fair Diffusion, to attenuate biases after the deployment of generative text-to-image models. Specifically, we demonstrate shifting a bias, based on human instructions, in any direction yielding arbitrarily new proportions for, e.g., identity groups. As our empirical evaluation demonstrates, this introduced control enables instructing generative image models on fairness, with no data filtering and additional training required.