Memory Attentive Fusion: External Language Model Integration for Transformer-based Sequence-to-Sequence Model

TL;DR

This paper introduces a memory attentive fusion method that effectively integrates external language models into Transformer-based seq2seq models, improving performance on text conversion tasks by leveraging attention mechanisms.

Contribution

It proposes a novel fusion technique that explicitly utilizes Transformer structures to incorporate external language models, unlike previous methods focused on RNN-based models.

Findings

Outperforms conventional fusion methods on text-style conversion tasks

Utilizes multi-hop attention to read memorized knowledge from external LM

Demonstrates the effectiveness of Transformer-specific fusion in seq2seq models

Abstract

This paper presents a novel fusion method for integrating an external language model (LM) into the Transformer based sequence-to-sequence (seq2seq) model. While paired data are basically required to train the seq2seq model, the external LM can be trained with only unpaired data. Thus, it is important to leverage memorized knowledge in the external LM for building the seq2seq model, since it is hard to prepare a large amount of paired data. However, the existing fusion methods assume that the LM is integrated with recurrent neural network-based seq2seq models instead of the Transformer. Therefore, this paper proposes a fusion method that can explicitly utilize network structures in the Transformer. The proposed method, called {\bf memory attentive fusion}, leverages the Transformer-style attention mechanism that repeats source-target attention in a multi-hop manner for reading the memorized knowledge in the LM. Our experiments on two text-style conversion tasks demonstrate that the proposed method performs better than conventional fusion methods.