Quantifying the Gaps Between Translation and Native Perception in Training for Multimodal, Multilingual Retrieval

TL;DR

This paper investigates the perceptual differences in multilingual vision-language models, quantifies performance gaps caused by translation methods, and explores caption augmentation strategies to improve model flexibility in multilingual retrieval tasks.

Contribution

It provides a case study quantifying perceptual gaps in multilingual models and evaluates caption augmentation to mitigate these gaps, highlighting areas for future research.

Findings

Performance gaps between native and translated captions quantified

Caption augmentation improves mean recall by +1.3

Gaps persist, indicating need for further research

Abstract

There is a scarcity of multilingual vision-language models that properly account for the perceptual differences that are reflected in image captions across languages and cultures. In this work, through a multimodal, multilingual retrieval case study, we quantify the existing lack of model flexibility. We empirically show performance gaps between training on captions that come from native German perception and captions that have been either machine-translated or human-translated from English into German. To address these gaps, we further propose and evaluate caption augmentation strategies. While we achieve mean recall improvements (+1.3), gaps still remain, indicating an open area of future work for the community.