TIBET: Identifying and Evaluating Biases in Text-to-Image Generative Models

TL;DR

This paper introduces TIBET, a novel method for identifying and quantifying biases in text-to-image generative models using counterfactual reasoning and semantic explanations, enhancing understanding of complex societal biases.

Contribution

The paper presents a general, automatic approach to detect and measure diverse biases in TTI models for any prompt, including intersectionality, with semantic explanations and validation through user studies.

Findings

Effective bias detection across multiple dimensions

Semantic explanations reveal complex bias interactions

User studies confirm alignment with human judgments

Abstract

Text-to-Image (TTI) generative models have shown great progress in the past few years in terms of their ability to generate complex and high-quality imagery. At the same time, these models have been shown to suffer from harmful biases, including exaggerated societal biases (e.g., gender, ethnicity), as well as incidental correlations that limit such a model's ability to generate more diverse imagery. In this paper, we propose a general approach to study and quantify a broad spectrum of biases, for any TTI model and for any prompt, using counterfactual reasoning. Unlike other works that evaluate generated images on a predefined set of bias axes, our approach automatically identifies potential biases that might be relevant to the given prompt, and measures those biases. In addition, we complement quantitative scores with post-hoc explanations in terms of semantic concepts in the images generated. We show that our method is uniquely capable of explaining complex multi-dimensional biases through semantic concepts, as well as the intersectionality between different biases for any given prompt. We perform extensive user studies to illustrate that the results of our method and analysis are consistent with human judgements.