Crossroads of Continents: Automated Artifact Extraction for Cultural Adaptation with Large Multimodal Models

TL;DR

This paper investigates the cultural understanding of large multimodal models using a new dataset, revealing disparities and stereotypes, and proposes a modular pipeline, CultureAdapt, to improve cultural representation in images.

Contribution

Introduces DalleStreet dataset, analyzes cultural associations in LMMs, and proposes CultureAdapt for cultural adaptation in images.

Findings

Disparities in cultural understanding across geographic regions.

Identification of stereotypes in model associations.

Demonstration of the effectiveness of CultureAdapt pipeline.

Abstract

We present a comprehensive three-phase study to examine (1) the cultural understanding of Large Multimodal Models (LMMs) by introducing DalleStreet, a large-scale dataset generated by DALL-E 3 and validated by humans, containing 9,935 images of 67 countries and 10 concept classes; (2) the underlying implicit and potentially stereotypical cultural associations with a cultural artifact extraction task; and (3) an approach to adapt cultural representation in an image based on extracted associations using a modular pipeline, CultureAdapt. We find disparities in cultural understanding at geographic sub-region levels with both open-source (LLaVA) and closed-source (GPT-4V) models on DalleStreet and other existing benchmarks, which we try to understand using over 18,000 artifacts that we identify in association to different countries. Our findings reveal a nuanced picture of the cultural competence of LMMs, highlighting the need to develop culture-aware systems. Dataset and code are available at https://github.com/iamshnoo/crossroads