On the Learning of Non-Autoregressive Transformers

TL;DR

This paper analyzes the challenges of training non-autoregressive Transformers for text generation, revealing how likelihood maximization affects token dependencies and proposing a unified framework to understand existing methods.

Contribution

It provides a theoretical and empirical analysis of NAT learning, unifies various training objectives, and offers insights to improve NAT training methods.

Findings

Likelihood training drops token dependencies, measured by conditional total correlation.

Existing objectives can be seen as likelihood maximization on proxy distributions.

The proposed perspective explains NAT learning phenomena and guides new training strategies.

Abstract

Non-autoregressive Transformer (NAT) is a family of text generation models, which aims to reduce the decoding latency by predicting the whole sentences in parallel. However, such latency reduction sacrifices the ability to capture left-to-right dependencies, thereby making NAT learning very challenging. In this paper, we present theoretical and empirical analyses to reveal the challenges of NAT learning and propose a unified perspective to understand existing successes. First, we show that simply training NAT by maximizing the likelihood can lead to an approximation of marginal distributions but drops all dependencies between tokens, where the dropped information can be measured by the dataset's conditional total correlation. Second, we formalize many previous objectives in a unified framework and show that their success can be concluded as maximizing the likelihood on a proxy distribution, leading to a reduced information loss. Empirical studies show that our perspective can explain the phenomena in NAT learning and guide the design of new training methods.