Semi-Autoregressive Training Improves Mask-Predict Decoding

TL;DR

This paper introduces SMART, a training method for masked language models that enhances semi-autoregressive decoding, leading to higher translation quality and closing the gap with autoregressive models.

Contribution

SMART is a novel training approach that incorporates model predictions into training data, improving semi-autoregressive translation performance.

Findings

Models trained with SMART achieve higher translation quality.

SMART reduces the performance gap between semi-autoregressive and autoregressive models.

Enhanced decoding efficiency with improved translation accuracy.

Abstract

The recently proposed mask-predict decoding algorithm has narrowed the performance gap between semi-autoregressive machine translation models and the traditional left-to-right approach. We introduce a new training method for conditional masked language models, SMART, which mimics the semi-autoregressive behavior of mask-predict, producing training examples that contain model predictions as part of their inputs. Models trained with SMART produce higher-quality translations when using mask-predict decoding, effectively closing the remaining performance gap with fully autoregressive models.