FairImagen: Post-Processing for Bias Mitigation in Text-to-Image Models

TL;DR

FairImagen is a post-processing framework that reduces societal biases in text-to-image models by manipulating prompt embeddings, improving fairness across demographic attributes without retraining the models.

Contribution

We propose a novel post-hoc debiasing method using Fair PCA and noise injection to mitigate biases in text-to-image diffusion models without altering their training.

Findings

Significantly reduces demographic biases in generated images.

Maintains reasonable image quality and prompt fidelity.

Outperforms existing post-hoc debiasing methods.

Abstract

Text-to-image diffusion models, such as Stable Diffusion, have demonstrated remarkable capabilities in generating high-quality and diverse images from natural language prompts. However, recent studies reveal that these models often replicate and amplify societal biases, particularly along demographic attributes like gender and race. In this paper, we introduce FairImagen (https://github.com/fuzihaofzh/FairImagen), a post-hoc debiasing framework that operates on prompt embeddings to mitigate such biases without retraining or modifying the underlying diffusion model. Our method integrates Fair Principal Component Analysis to project CLIP-based input embeddings into a subspace that minimizes group-specific information while preserving semantic content. We further enhance debiasing effectiveness through empirical noise injection and propose a unified cross-demographic projection method that enables simultaneous debiasing across multiple demographic attributes. Extensive experiments across gender, race, and intersectional settings demonstrate that FairImagen significantly improves fairness with a moderate trade-off in image quality and prompt fidelity. Our framework outperforms existing post-hoc methods and offers a simple, scalable, and model-agnostic solution for equitable text-to-image generation.