Culture in Action: Evaluating Text-to-Image Models through Social Activities

TL;DR

This paper introduces CULTIVate, a comprehensive benchmark for evaluating text-to-image models on their ability to accurately depict cross-cultural social activities, highlighting disparities and biases in current models.

Contribution

The paper presents CULTIVate, a new benchmark with metrics for assessing cultural faithfulness in T2I models across diverse social activities and regions, addressing gaps in existing evaluations.

Findings

Models perform better for global north countries.

Systematic disparities and failure modes identified.

Metrics correlate strongly with human judgments.

Abstract

Text-to-image (T2I) diffusion models achieve impressive photorealism by training on large-scale web data, but models inherit cultural biases and fail to depict underrepresented regions faithfully. Existing cultural benchmarks focus mainly on object-centric categories (e.g., food, attire, and architecture), overlooking the social and daily activities that more clearly reflect cultural norms. Few metrics exist for measuring cultural faithfulness. We introduce CULTIVate, a benchmark for evaluating T2I models on cross-cultural activities (e.g., greetings, dining, games, traditional dances, and cultural celebrations). CULTIVate spans 16 countries with 576 prompts and more than 19,000 images, and provides an explainable descriptor-based evaluation framework across multiple cultural dimensions, including background, attire, objects, and interactions. We propose four metrics to measure cultural alignment, hallucination, exaggerated elements, and diversity. Our findings reveal systematic disparities: models perform better for global north countries than for the global south, with distinct failure modes across T2I systems. Human studies confirm that our metrics correlate more strongly with human judgments than existing text-image metrics.