FAIntbench: A Holistic and Precise Benchmark for Bias Evaluation in Text-to-Image Models

TL;DR

FAIntbench is a comprehensive benchmark designed to evaluate biases in Text-to-Image models across multiple dimensions, aiding in the development of more fair and unbiased AI systems.

Contribution

This paper introduces FAIntbench, a holistic and precise bias evaluation framework for T2I models, addressing limitations of existing benchmarks.

Findings

FAIntbench effectively identifies various biases in T2I models.

Evaluation of seven large-scale T2I models demonstrates the benchmark's utility.

Reveals new research questions about bias side-effects, such as distillation.

Abstract

The rapid development and reduced barriers to entry for Text-to-Image (T2I) models have raised concerns about the biases in their outputs, but existing research lacks a holistic definition and evaluation framework of biases, limiting the enhancement of debiasing techniques. To address this issue, we introduce FAIntbench, a holistic and precise benchmark for biases in T2I models. In contrast to existing benchmarks that evaluate bias in limited aspects, FAIntbench evaluate biases from four dimensions: manifestation of bias, visibility of bias, acquired attributes, and protected attributes. We applied FAIntbench to evaluate seven recent large-scale T2I models and conducted human evaluation, whose results demonstrated the effectiveness of FAIntbench in identifying various biases. Our study also revealed new research questions about biases, including the side-effect of distillation. The findings presented here are preliminary, highlighting the potential of FAIntbench to advance future research aimed at mitigating the biases in T2I models. Our benchmark is publicly available to ensure the reproducibility.