MoESD: Mixture of Experts Stable Diffusion to Mitigate Gender Bias

TL;DR

This paper introduces MoESD, a novel method using a mixture of experts and bias adapters to reduce gender bias in text-to-image models, specifically targeting biases present in the text encoder, while preserving image quality.

Contribution

The paper presents a new approach called MoESD with Bias Adapters that effectively mitigates gender bias in text-to-image models by identifying and addressing bias in the text encoder.

Findings

Successfully reduces gender bias in generated images.

Maintains high image quality after bias mitigation.

Highlights importance of special tokens during mitigation.

Abstract

Text-to-image models are known to propagate social biases. For example, when prompted to generate images of people in certain professions, these models tend to systematically generate specific genders or ethnicities. In this paper, we show that this bias is already present in the text encoder of the model and introduce a Mixture-of-Experts approach by identifying text-encoded bias in the latent space and then creating a Bias-Identification Gate mechanism. More specifically, we propose MoESD (Mixture of Experts Stable Diffusion) with BiAs (Bias Adapters) to mitigate gender bias in text-to-image models. We also demonstrate that introducing an arbitrary special token to the prompt is essential during the mitigation process. With experiments focusing on gender bias, we show that our approach successfully mitigates gender bias while maintaining image quality.