Evaluation of Cultural Competence of Vision-Language Models

TL;DR

This paper highlights the limitations of current vision-language models in understanding cultural nuances and proposes a comprehensive framework based on visual culture studies to evaluate their cultural competence.

Contribution

It introduces five new frameworks derived from cultural studies to systematically analyze and improve VLMs' cultural understanding.

Findings

Current VLMs lack comprehensive cultural competence

Proposed frameworks enable systematic cultural analysis of images

Foundation for future improvements in culturally aware VLMs

Abstract

Modern vision-language models (VLMs) often fail at cultural competency evaluations and benchmarks. Given the diversity of applications built upon VLMs, there is renewed interest in understanding how they encode cultural nuances. While individual aspects of this problem have been studied, we still lack a comprehensive framework for systematically identifying and annotating the nuanced cultural dimensions present in images for VLMs. This position paper argues that foundational methodologies from visual culture studies (cultural studies, semiotics, and visual studies) are necessary for cultural analysis of images. Building upon this review, we propose a set of five frameworks, corresponding to cultural dimensions, that must be considered for a more complete analysis of the cultural competencies of VLMs.