RS-Corrector: Correcting the Racial Stereotypes in Latent Diffusion Models

TL;DR

This paper introduces RS-Corrector, a method to reduce racial stereotypes in latent diffusion models during inference, improving fairness without altering the original model.

Contribution

The paper proposes RS-Corrector, a novel inference-stage correction framework that mitigates racial stereotypes in diffusion models without fine-tuning.

Findings

Effectively reduces racial stereotypes in generated images.

Maintains original model performance while improving fairness.

Works during inference without additional training.

Abstract

Recent text-conditioned image generation models have demonstrated an exceptional capacity to produce diverse and creative imagery with high visual quality. However, when pre-trained on billion-sized datasets randomly collected from the Internet, where potential biased human preferences exist, these models tend to produce images with common and recurring stereotypes, particularly for certain racial groups. In this paper, we conduct an initial analysis of the publicly available Stable Diffusion model and its derivatives, highlighting the presence of racial stereotypes. These models often generate distorted or biased images for certain racial groups, emphasizing stereotypical characteristics. To address these issues, we propose a framework called "RS-Corrector", designed to establish an anti-stereotypical preference in the latent space and update the latent code for refined generated results. The correction process occurs during the inference stage without requiring fine-tuning of the original model. Extensive empirical evaluations demonstrate that the introduced \themodel effectively corrects the racial stereotypes of the well-trained Stable Diffusion model while leaving the original model unchanged.