Navigating Cultural Chasms: Exploring and Unlocking the Cultural POV of Text-To-Image Models

TL;DR

This paper investigates the cultural perceptions embedded in multilingual Text-To-Image models, introducing new evaluation methods and a diverse dataset to understand and unlock their cultural knowledge for cross-cultural applications.

Contribution

It presents a hierarchical ontology of culture in TTI models, develops prompt templates for cultural knowledge extraction, and introduces the CulText2I dataset for comprehensive evaluation.

Findings

Cultural encoding varies across models and languages.

Prompt templates effectively reveal cultural concepts.

Evaluation techniques provide nuanced insights into cultural content.

Abstract

Text-To-Image (TTI) models, such as DALL-E and StableDiffusion, have demonstrated remarkable prompt-based image generation capabilities. Multilingual encoders may have a substantial impact on the cultural agency of these models, as language is a conduit of culture. In this study, we explore the cultural perception embedded in TTI models by characterizing culture across three hierarchical tiers: cultural dimensions, cultural domains, and cultural concepts. Based on this ontology, we derive prompt templates to unlock the cultural knowledge in TTI models, and propose a comprehensive suite of evaluation techniques, including intrinsic evaluations using the CLIP space, extrinsic evaluations with a Visual-Question-Answer (VQA) model and human assessments, to evaluate the cultural content of TTI-generated images. To bolster our research, we introduce the CulText2I dataset, derived from six diverse TTI models and spanning ten languages. Our experiments provide insights regarding Do, What, Which and How research questions about the nature of cultural encoding in TTI models, paving the way for cross-cultural applications of these models.