Fair Generation without Unfair Distortions: Debiasing Text-to-Image Generation with Entanglement-Free Attention

TL;DR

This paper introduces Entanglement-Free Attention (EFA), a novel method for bias mitigation in text-to-image models that effectively reduces societal biases without distorting non-target attributes, ensuring fairer image generation.

Contribution

The paper proposes EFA, a new attention mechanism that disentangles target and non-target attributes during bias mitigation in diffusion-based T2I models.

Findings

EFA outperforms existing bias mitigation methods in reducing societal biases.

EFA preserves non-target attributes and the original distribution of generated images.

Extensive experiments validate the effectiveness of EFA in fair image generation.

Abstract

Recent advancements in diffusion-based text-to-image (T2I) models have enabled the generation of high-quality and photorealistic images from text. However, they often exhibit societal biases related to gender, race, and socioeconomic status, thereby potentially reinforcing harmful stereotypes and shaping public perception in unintended ways. While existing bias mitigation methods demonstrate effectiveness, they often encounter attribute entanglement, where adjustments to attributes relevant to the bias (i.e., target attributes) unintentionally alter attributes unassociated with the bias (i.e., non-target attributes), causing undesirable distribution shifts. To address this challenge, we introduce Entanglement-Free Attention (EFA), a method that accurately incorporates target attributes (e.g., White, Black, and Asian) while preserving non-target attributes (e.g., background) during bias mitigation. At inference time, EFA randomly samples a target attribute with equal probability and adjusts the cross-attention in selected layers to incorporate the sampled attribute, achieving a fair distribution of target attributes. Extensive experiments demonstrate that EFA outperforms existing methods in mitigating bias while preserving non-target attributes, thereby maintaining the original model's output distribution and generative capacity.