Parallel Attention Forcing for Machine Translation

TL;DR

This paper introduces parallel attention forcing with scheduled control for training Transformer-based and RNN models in machine translation, addressing training mismatches and improving performance.

Contribution

It proposes two extensions of attention forcing—scheduled attention forcing and parallel attention forcing—for better training of sequence-to-sequence models with discrete outputs.

Findings

Improved translation quality with attention forcing methods.

Parallel attention forcing enables efficient training of Transformer models.

Scheduled attention forcing effectively manages attention guidance during training.

Abstract

Attention-based autoregressive models have achieved state-of-the-art performance in various sequence-to-sequence tasks, including Text-To-Speech (TTS) and Neural Machine Translation (NMT), but can be difficult to train. The standard training approach, teacher forcing, guides a model with the reference back-history. During inference, the generated back-history must be used. This mismatch limits the evaluation performance. Attention forcing has been introduced to address the mismatch, guiding the model with the generated back-history and reference attention. While successful in tasks with continuous outputs like TTS, attention forcing faces additional challenges in tasks with discrete outputs like NMT. This paper introduces the two extensions of attention forcing to tackle these challenges. (1) Scheduled attention forcing automatically turns attention forcing on and off, which is essential for tasks with discrete outputs. (2) Parallel attention forcing makes training parallel, and is applicable to Transformer-based models. The experiments show that the proposed approaches improve the performance of models based on RNNs and Transformers.