Transfer Learning for Sequence Generation: from Single-source to Multi-source

TL;DR

This paper introduces a two-stage finetuning approach and a novel MSG model with a fine encoder to improve multi-source sequence generation tasks, achieving state-of-the-art results across multiple benchmarks.

Contribution

The paper proposes a two-stage finetuning method and a new MSG model with a fine encoder to better utilize pretrained models for multi-source sequence generation.

Findings

Achieves new state-of-the-art on WMT17 APE and multi-source translation tasks.

Outperforms strong baselines in document-level translation.

Effectively alleviates catastrophic forgetting in MSG tasks.

Abstract

Multi-source sequence generation (MSG) is an important kind of sequence generation tasks that takes multiple sources, including automatic post-editing, multi-source translation, multi-document summarization, etc. As MSG tasks suffer from the data scarcity problem and recent pretrained models have been proven to be effective for low-resource downstream tasks, transferring pretrained sequence-to-sequence models to MSG tasks is essential. Although directly finetuning pretrained models on MSG tasks and concatenating multiple sources into a single long sequence is regarded as a simple method to transfer pretrained models to MSG tasks, we conjecture that the direct finetuning method leads to catastrophic forgetting and solely relying on pretrained self-attention layers to capture cross-source information is not sufficient. Therefore, we propose a two-stage finetuning method to alleviate the pretrain-finetune discrepancy and introduce a novel MSG model with a fine encoder to learn better representations in MSG tasks. Experiments show that our approach achieves new state-of-the-art results on the WMT17 APE task and multi-source translation task using the WMT14 test set. When adapted to document-level translation, our framework outperforms strong baselines significantly.