Model-Agnostic Gender Bias Control for Text-to-Image Generation via Sparse Autoencoder

TL;DR

SAE Debias is a lightweight, model-agnostic framework that uses sparse autoencoders to identify and suppress gender bias in text-to-image models, improving fairness without retraining or architectural changes.

Contribution

This work introduces SAE Debias, the first application of sparse autoencoders for debiasing in T2I models, providing interpretable and reusable bias control across multiple models.

Findings

Significantly reduces gender bias in T2I outputs.

Maintains high image generation quality.

Operates without retraining or model-specific adjustments.

Abstract

Text-to-image (T2I) diffusion models often exhibit gender bias, particularly by generating stereotypical associations between professions and gendered subjects. This paper presents SAE Debias, a lightweight and model-agnostic framework for mitigating such bias in T2I generation. Unlike prior approaches that rely on CLIP-based filtering or prompt engineering, which often require model-specific adjustments and offer limited control, SAE Debias operates directly within the feature space without retraining or architectural modifications. By leveraging a k-sparse autoencoder pre-trained on a gender bias dataset, the method identifies gender-relevant directions within the sparse latent space, capturing professional stereotypes. Specifically, a biased direction per profession is constructed from sparse latents and suppressed during inference to steer generations toward more gender-balanced outputs. Trained only once, the sparse autoencoder provides a reusable debiasing direction, offering effective control and interpretable insight into biased subspaces. Extensive evaluations across multiple T2I models, including Stable Diffusion 1.4, 1.5, 2.1, and SDXL, demonstrate that SAE Debias substantially reduces gender bias while preserving generation quality. To the best of our knowledge, this is the first work to apply sparse autoencoders for identifying and intervening in gender bias within T2I models. These findings contribute toward building socially responsible generative AI, providing an interpretable and model-agnostic tool to support fairness in text-to-image generation.