CaMMT: Benchmarking Culturally Aware Multimodal Machine Translation

TL;DR

This paper introduces CaMMT, a benchmark dataset with images and captions in multiple languages, to evaluate how visual context can improve culturally aware machine translation, especially for region-specific content.

Contribution

The paper presents CaMMT, a new benchmark dataset for multimodal translation that incorporates cultural context via images, and evaluates vision-language models on this dataset.

Findings

Visual context improves translation quality for culturally-specific items.

Images help disambiguate meanings and improve gender accuracy.

Multimodal models outperform text-only models in cultural translation tasks.

Abstract

Translating cultural content poses challenges for machine translation systems due to the differences in conceptualizations between cultures, where language alone may fail to convey sufficient context to capture region-specific meanings. In this work, we investigate whether images can act as cultural context in multimodal translation. We introduce CaMMT, a human-curated benchmark of over 5,800 triples of images along with parallel captions in English and regional languages. Using this dataset, we evaluate five Vision Language Models (VLMs) in text-only and text+image settings. Through automatic and human evaluations, we find that visual context generally improves translation quality, especially in handling Culturally-Specific Items (CSIs), disambiguation, and correct gender marking. By releasing CaMMT, our objective is to support broader efforts to build and evaluate multimodal translation systems that are better aligned with cultural nuance and regional variations.