The MeMAD Submission to the WMT18 Multimodal Translation Task

TL;DR

This paper presents the MeMAD system for the WMT18 multimodal translation task, adapting Transformer models to include visual features, but finds text quality and data more impactful than visual input.

Contribution

It introduces a Transformer-based multimodal translation approach and demonstrates the importance of text data quality over visual features.

Findings

Visual features have limited impact on translation quality.

High-quality text-only NMT systems outperform multimodal variants.

Additional data improves translation performance.

Abstract

This paper describes the MeMAD project entry to the WMT Multimodal Machine Translation Shared Task. We propose adapting the Transformer neural machine translation (NMT) architecture to a multi-modal setting. In this paper, we also describe the preliminary experiments with text-only translation systems leading us up to this choice. We have the top scoring system for both English-to-German and English-to-French, according to the automatic metrics for flickr18. Our experiments show that the effect of the visual features in our system is small. Our largest gains come from the quality of the underlying text-only NMT system. We find that appropriate use of additional data is effective.