Does Multimodality Help Human and Machine for Translation and Image Captioning?

TL;DR

This paper investigates whether multimodal data improves translation and image captioning by comparing different systems and evaluating both automatic metrics and human judgment, demonstrating that multimodal approaches yield superior results.

Contribution

It introduces and compares multimodal and monomodal systems for translation and captioning, showing the benefits of multimodal data through comprehensive evaluation.

Findings

Multimodal systems outperform monomodal ones in BLEU and METEOR scores.

Human evaluation indicates multimodal data enhances translation and captioning quality.

The best results were achieved by systems using multimodal data.

Abstract

This paper presents the systems developed by LIUM and CVC for the WMT16 Multimodal Machine Translation challenge. We explored various comparative methods, namely phrase-based systems and attentional recurrent neural networks models trained using monomodal or multimodal data. We also performed a human evaluation in order to estimate the usefulness of multimodal data for human machine translation and image description generation. Our systems obtained the best results for both tasks according to the automatic evaluation metrics BLEU and METEOR.