Ensemble Sequence Level Training for Multimodal MT: OSU-Baidu WMT18 Multimodal Machine Translation System Report

TL;DR

This paper presents a multimodal machine translation system that integrates image features into the decoder, explores sequence-level training methods, and ensembles multiple models to achieve top performance in WMT 2018 tasks.

Contribution

It introduces a simple image feature integration method and explores sequence-level training techniques for improved multimodal translation performance.

Findings

Image features fed to decoder improve translation quality

Sequence-level training methods enhance system performance

Ensembling diverse models yields top results in WMT 2018 tasks

Abstract

This paper describes multimodal machine translation systems developed jointly by Oregon State University and Baidu Research for WMT 2018 Shared Task on multimodal translation. In this paper, we introduce a simple approach to incorporate image information by feeding image features to the decoder side. We also explore different sequence level training methods including scheduled sampling and reinforcement learning which lead to substantial improvements. Our systems ensemble several models using different architectures and training methods and achieve the best performance for three subtasks: En-De and En-Cs in task 1 and (En+De+Fr)-Cs task 1B.