T2IAT: Measuring Valence and Stereotypical Biases in Text-to-Image Generation

TL;DR

This paper introduces T2IAT, a framework inspired by social psychology's IAT, to measure complex human biases and stereotypes in text-to-image generative models, revealing their implicit stereotypical behaviors.

Contribution

The paper presents a novel T2IAT framework for quantifying implicit stereotypes in text-to-image models, extending bias measurement to complex social attributes and valence associations.

Findings

Detected stereotypes related to gender and skin tone in image generation

Replicated bias tests on diverse concepts like flowers and insects

Confirmed presence of complex stereotypical behaviors in models

Abstract

Warning: This paper contains several contents that may be toxic, harmful, or offensive. In the last few years, text-to-image generative models have gained remarkable success in generating images with unprecedented quality accompanied by a breakthrough of inference speed. Despite their rapid progress, human biases that manifest in the training examples, particularly with regard to common stereotypical biases, like gender and skin tone, still have been found in these generative models. In this work, we seek to measure more complex human biases exist in the task of text-to-image generations. Inspired by the well-known Implicit Association Test (IAT) from social psychology, we propose a novel Text-to-Image Association Test (T2IAT) framework that quantifies the implicit stereotypes between concepts and valence, and those in the images. We replicate the previously documented bias tests on generative models, including morally neutral tests on flowers and insects as well as demographic stereotypical tests on diverse social attributes. The results of these experiments demonstrate the presence of complex stereotypical behaviors in image generations.