Debiasing Text-to-Image Diffusion Models

TL;DR

This paper addresses social bias in Text-to-Image diffusion models by proposing an iterative distribution alignment method that effectively reduces bias with fast convergence.

Contribution

It introduces an iterative distribution alignment approach to mitigate social bias in diffusion models, improving efficiency over previous reinforcement-based methods.

Findings

IDA method achieves fast convergence

Effectively reduces social bias in TTI models

Outperforms reinforcement-based approaches

Abstract

Learning-based Text-to-Image (TTI) models like Stable Diffusion have revolutionized the way visual content is generated in various domains. However, recent research has shown that nonnegligible social bias exists in current state-of-the-art TTI systems, which raises important concerns. In this work, we target resolving the social bias in TTI diffusion models. We begin by formalizing the problem setting and use the text descriptions of bias groups to establish an unsafe direction for guiding the diffusion process. Next, we simplify the problem into a weight optimization problem and attempt a Reinforcement solver, Policy Gradient, which shows sub-optimal performance with slow convergence. Further, to overcome limitations, we propose an iterative distribution alignment (IDA) method. Despite its simplicity, we show that IDA shows efficiency and fast convergence in resolving the social bias in TTI diffusion models. Our code will be released.