A Multimodal Dataset for Visually Grounded Ambiguity in Machine Translation

TL;DR

This paper introduces VIDA, a new dataset with 2,500 instances for evaluating visually grounded ambiguity resolution in machine translation, along with metrics and experiments demonstrating the effectiveness of chain-of-thought fine-tuning.

Contribution

The paper presents VIDA, a carefully curated dataset for visual ambiguity in translation, and proposes disambiguation-centric metrics and methods to improve ambiguity resolution.

Findings

Chain-of-thought fine-tuning improves disambiguation accuracy.

VIDA dataset reveals challenges in visual ambiguity resolution.

CoT-SFT generalizes better on out-of-distribution data.

Abstract

Ambiguity resolution is a key challenge in multimodal machine translation (MMT), where models must genuinely leverage visual input to map an ambiguous expression to its intended meaning. Although prior work has proposed disambiguation-oriented benchmarks that provide supportive evidence for the role of vision, we observe substantial issues in data quality and a mismatch with translation scenarios. Moreover, existing ambiguity-oriented evaluations are not well suited to broader ambiguity types in open-ended translation. To address these limitations, we present VIDA (Visually-Dependent Ambiguity), a dataset of 2,500 carefully curated instances in which resolving an annotated ambiguous source span requires visual evidence. We further propose Disambiguation-Centric Metrics that use an LLM-as-a-judge classifier to verify whether annotated ambiguous expressions are resolved correctly at the span level. Experiments with two state-of-the-art Large Vision Language Models under vanilla inference, supervised fine-tuning (SFT), and our chain-of-thought SFT (CoT-SFT) show that while SFT improves overall translation quality, CoT-SFT yields more consistent gains in disambiguation accuracy, especially on out-of-distribution subsets, indicating a stronger generalization for resolving diverse ambiguity types.