Beyond Words: Exploring Cultural Value Sensitivity in Multimodal Models

TL;DR

This paper evaluates how large vision-language models (VLMs) reflect cultural values, revealing that their alignment with cultural sensitivities varies with context, highlighting challenges and potential for improvement.

Contribution

It is the first comprehensive study assessing cultural value sensitivity in multimodal models, comparing their performance to language-only models across different scales.

Findings

VLMs exhibit cultural value sensitivity similar to LLMs.

Performance in aligning with cultural values is highly context-dependent.

Using images can enhance understanding of cultural values but introduces variability.

Abstract

Investigating value alignment in Large Language Models (LLMs) based on cultural context has become a critical area of research. However, similar biases have not been extensively explored in large vision-language models (VLMs). As the scale of multimodal models continues to grow, it becomes increasingly important to assess whether images can serve as reliable proxies for culture and how these values are embedded through the integration of both visual and textual data. In this paper, we conduct a thorough evaluation of multimodal model at different scales, focusing on their alignment with cultural values. Our findings reveal that, much like LLMs, VLMs exhibit sensitivity to cultural values, but their performance in aligning with these values is highly context-dependent. While VLMs show potential in improving value understanding through the use of images, this alignment varies significantly across contexts highlighting the complexities and underexplored challenges in the alignment of multimodal models.