A Framework for Benchmarking Fairness-Utility Trade-offs in Text-to-Image Models via Pareto Frontiers

TL;DR

This paper introduces a Pareto frontier-based framework for benchmarking fairness-utility trade-offs in text-to-image models, enabling systematic and reproducible evaluation of debiasing methods across different models and hyperparameters.

Contribution

It proposes a novel evaluation method using Pareto frontiers to compare fairness and utility in text-to-image models, improving upon subjective and non-reproducible previous approaches.

Findings

Most default hyperparameters are dominated solutions in fairness-utility space

Better hyperparameters can be easily identified for improved fairness or utility

The method enables comparison across different models and debiasing techniques

Abstract

Achieving fairness in text-to-image generation demands mitigating social biases without compromising visual fidelity, a challenge critical to responsible AI. Current fairness evaluation procedures for text-to-image models rely on qualitative judgment or narrow comparisons, which limit the capacity to assess both fairness and utility in these models and prevent reproducible assessment of debiasing methods. Existing approaches typically employ ad-hoc, human-centered visual inspections that are both error-prone and difficult to replicate. We propose a method for evaluating fairness and utility in text-to-image models using Pareto-optimal frontiers across hyperparametrization of debiasing methods. Our method allows for comparison between distinct text-to-image models, outlining all configurations that optimize fairness for a given utility and vice-versa. To illustrate our evaluation method, we use Normalized Shannon Entropy and ClipScore for fairness and utility evaluation, respectively. We assess fairness and utility in Stable Diffusion, Fair Diffusion, SDXL, DeCoDi, and FLUX text-to-image models. Our method shows that most default hyperparameterizations of the text-to-image model are dominated solutions in the fairness-utility space, and it is straightforward to find better hyperparameters.