Dynamic Fusion: Attentional Language Model for Neural Machine Translation

TL;DR

This paper introduces Dynamic Fusion, an attentive mechanism that effectively integrates language models into neural machine translation, improving translation quality by dynamically considering translation history and grammatical structure.

Contribution

The work proposes a novel attentive fusion approach that adaptively combines language and translation models, addressing limitations of previous static weighting methods.

Findings

Improved BLEU and RIBES scores in English-Japanese translation

Enhanced grammatical conformity in language modeling

Dynamic fusion outperforms previous integration methods

Abstract

Neural Machine Translation (NMT) can be used to generate fluent output. As such, language models have been investigated for incorporation with NMT. In prior investigations, two models have been used: a translation model and a language model. The translation model's predictions are weighted by the language model with a hand-crafted ratio in advance. However, these approaches fail to adopt the language model weighting with regard to the translation history. In another line of approach, language model prediction is incorporated into the translation model by jointly considering source and target information. However, this line of approach is limited because it largely ignores the adequacy of the translation output. Accordingly, this work employs two mechanisms, the translation model and the language model, with an attentive architecture to the language model as an auxiliary element of the translation model. Compared with previous work in English--Japanese machine translation using a language model, the experimental results obtained with the proposed Dynamic Fusion mechanism improve BLEU and Rank-based Intuitive Bilingual Evaluation Scores (RIBES) scores. Additionally, in the analyses of the attention and predictivity of the language model, the Dynamic Fusion mechanism allows predictive language modeling that conforms to the appropriate grammatical structure.