Deliberation Networks and How to Train Them

TL;DR

This paper presents a comprehensive framework for training deliberation networks, clarifying best practices and options for different tasks, and simplifying the training process while improving performance.

Contribution

It introduces a unifying framework for training deliberation networks, addressing key questions and providing guidelines for various scenarios and tasks.

Findings

Gradient approximation is generally simpler.

Separate training is preferable for parallel training.

Use free running mode for intermediate models.

Abstract

Deliberation networks are a family of sequence-to-sequence models, which have achieved state-of-the-art performance in a wide range of tasks such as machine translation and speech synthesis. A deliberation network consists of multiple standard sequence-to-sequence models, each one conditioned on the initial input and the output of the previous model. During training, there are several key questions: whether to apply Monte Carlo approximation to the gradients or the loss, whether to train the standard models jointly or separately, whether to run an intermediate model in teacher forcing or free running mode, whether to apply task-specific techniques. Previous work on deliberation networks typically explores one or two training options for a specific task. This work introduces a unifying framework, covering various training options, and addresses the above questions. In general, it is simpler to approximate the gradients. When parallel training is essential, separate training should be adopted. Regardless of the task, the intermediate model should be in free running mode. For tasks where the output is continuous, a guided attention loss can be used to prevent degradation into a standard model.