Evaluating Perspectival Biases in Cross-Modal Retrieval

TL;DR

This paper investigates how cultural and linguistic biases influence multimodal retrieval systems, revealing systematic biases and proposing the need for strategies to decouple language from culture for fairer retrieval outcomes.

Contribution

Introduces the 3XCM benchmark to isolate cultural and linguistic biases and analyzes their impact on cross-modal retrieval performance.

Findings

Models favor prevalent languages over semantically faithful entries.

A 'tugging effect' exists between semantic alignment and cultural association.

Biases are more pronounced in low-resource languages with insufficient semantic resolution.

Abstract

Multimodal retrieval systems are expected to operate in a semantic space, agnostic to the language or cultural origin of the query. In practice, however, retrieval outcomes systematically reflect perspectival biases: deviations shaped by linguistic prevalence and cultural associations. We introduce the Cross-Cultural, Cross-Modal, Cross-lingual Multimodal (3XCM) benchmark to isolate these effects. Results from our studies indicate that, for image-to-text retrieval, models tend to favor entries from prevalent languages over those that are semantically faithful. For text-to-image retrieval, we observe a consistent "tugging effect" in the joint embedding space between semantic alignment and language-conditioned cultural association. When semantic representations are insufficiently resolved, particularly in low-resource languages, similarity is increasingly governed by culturally familiar visual patterns, leading to systematic association bias in retrieval. Our findings suggest that achieving equitable multimodal retrieval necessitates targeted strategies that explicitly decouple language from culture, rather than relying solely on broader data exposure. This work highlights the need to treat linguistic and cultural biases as distinct, measurable challenges in multimodal representation learning.