DIG In: Evaluating Disparities in Image Generations with Indicators for Geographic Diversity

TL;DR

This paper introduces three indicators to evaluate geographic disparities in text-to-image generative systems, revealing biases and trade-offs in realism, diversity, and consistency across regions, especially for Africa and West Asia.

Contribution

The paper presents novel indicators for automatically benchmarking geographic biases in image generation models, enabling responsible assessment of their real-world diversity.

Findings

Models show less realism and diversity for Africa and West Asia.

Prompting with geographic info reduces image diversity and consistency.

Models exhibit more regional disparities for certain objects.

Abstract

The unprecedented photorealistic results achieved by recent text-to-image generative systems and their increasing use as plug-and-play content creation solutions make it crucial to understand their potential biases. In this work, we introduce three indicators to evaluate the realism, diversity and prompt-generation consistency of text-to-image generative systems when prompted to generate objects from across the world. Our indicators complement qualitative analysis of the broader impact of such systems by enabling automatic and efficient benchmarking of geographic disparities, an important step towards building responsible visual content creation systems. We use our proposed indicators to analyze potential geographic biases in state-of-the-art visual content creation systems and find that: (1) models have less realism and diversity of generations when prompting for Africa and West Asia than Europe, (2) prompting with geographic information comes at a cost to prompt-consistency and diversity of generated images, and (3) models exhibit more region-level disparities for some objects than others. Perhaps most interestingly, our indicators suggest that progress in image generation quality has come at the cost of real-world geographic representation. Our comprehensive evaluation constitutes a crucial step towards ensuring a positive experience of visual content creation for everyone.