Culture Affordance Atlas: Reconciling Object Diversity Through Functional Mapping

TL;DR

This paper introduces the Culture Affordance Atlas, a culturally grounded object categorization framework that reduces socioeconomic biases in vision-language models, enhancing their fairness and effectiveness across diverse cultural contexts.

Contribution

We propose a function-centric framework and create the Culture Affordance Atlas, significantly reducing socioeconomic performance gaps in VL models and identifying culturally essential objects often overlooked.

Findings

Function-centric labels reduce socioeconomic gaps by median 6 percentage points.

The Atlas includes 46 functions and 288 objects, improving cultural inclusivity.

Empirical analysis with CLIP shows enhanced model fairness.

Abstract

Culture shapes the objects people use and for what purposes, yet mainstream Vision-Language (VL) datasets frequently exhibit cultural biases, disproportionately favoring higher-income, Western contexts. This imbalance reduces model generalizability and perpetuates performance disparities, especially impacting lower-income and non-Western communities. To address these disparities, we propose a novel function-centric framework that categorizes objects by the functions they fulfill, across diverse cultural and economic contexts. We implement this framework by creating the Culture Affordance Atlas, a re-annotated and culturally grounded restructuring of the Dollar Street dataset spanning 46 functions and 288 objects publicly available at https://lit.eecs.umich.edu/CultureAffordance-Atlas/index.html. Through extensive empirical analyses using the CLIP model, we demonstrate that function-centric labels substantially reduce socioeconomic performance gaps between high- and low-income groups by a median of 6 pp (statistically significant), improving model effectiveness for lower-income contexts. Furthermore, our analyses reveals numerous culturally essential objects that are frequently overlooked in prominent VL datasets. Our contributions offer a scalable pathway toward building inclusive VL datasets and equitable AI systems.