FairRAG: Fair Human Generation via Fair Retrieval Augmentation

TL;DR

FairRAG is a new framework that improves fairness in human image generation by conditioning pre-trained models on diverse reference images retrieved from an external database, effectively reducing bias.

Contribution

It introduces a novel retrieval-augmented approach with a lightweight conditioning module and debiasing strategies to enhance demographic diversity in generated images.

Findings

Outperforms existing methods in demographic diversity

Maintains high image-text alignment and fidelity

Incur minimal computational overhead

Abstract

Existing text-to-image generative models reflect or even amplify societal biases ingrained in their training data. This is especially concerning for human image generation where models are biased against certain demographic groups. Existing attempts to rectify this issue are hindered by the inherent limitations of the pre-trained models and fail to substantially improve demographic diversity. In this work, we introduce Fair Retrieval Augmented Generation (FairRAG), a novel framework that conditions pre-trained generative models on reference images retrieved from an external image database to improve fairness in human generation. FairRAG enables conditioning through a lightweight linear module that projects reference images into the textual space. To enhance fairness, FairRAG applies simple-yet-effective debiasing strategies, providing images from diverse demographic groups during the generative process. Extensive experiments demonstrate that FairRAG outperforms existing methods in terms of demographic diversity, image-text alignment, and image fidelity while incurring minimal computational overhead during inference.