VISA: An Ambiguous Subtitles Dataset for Visual Scene-Aware Machine Translation

TL;DR

VISA is a new dataset of 40,000 Japanese-English movie subtitles with ambiguous source sentences and corresponding video clips, designed to challenge and advance multimodal machine translation research.

Contribution

The paper introduces VISA, a novel dataset with ambiguous subtitles and video context, addressing limitations of existing datasets for multimodal machine translation.

Findings

VISA is challenging for current MMT systems.

The dataset includes ambiguity types: Polysemy and Omission.

VISA facilitates research on context-aware translation.

Abstract

Existing multimodal machine translation (MMT) datasets consist of images and video captions or general subtitles, which rarely contain linguistic ambiguity, making visual information not so effective to generate appropriate translations. We introduce VISA, a new dataset that consists of 40k Japanese-English parallel sentence pairs and corresponding video clips with the following key features: (1) the parallel sentences are subtitles from movies and TV episodes; (2) the source subtitles are ambiguous, which means they have multiple possible translations with different meanings; (3) we divide the dataset into Polysemy and Omission according to the cause of ambiguity. We show that VISA is challenging for the latest MMT system, and we hope that the dataset can facilitate MMT research. The VISA dataset is available at: https://github.com/ku-nlp/VISA.