The Volctrans GLAT System: Non-autoregressive Translation Meets WMT21

TL;DR

This paper introduces a non-autoregressive translation system based on the Glancing Transformer, achieving state-of-the-art BLEU scores in German-English translation at WMT21, with faster decoding than autoregressive models.

Contribution

First practical non-autoregressive translation system scaled for WMT competition, outperforming autoregressive models in BLEU score.

Findings

Achieved BLEU score of 35.0 on German-English translation

Enabled fast parallel decoding with high accuracy

Outperformed all strong autoregressive models in the WMT21 task

Abstract

This paper describes the Volctrans' submission to the WMT21 news translation shared task for German->English translation. We build a parallel (i.e., non-autoregressive) translation system using the Glancing Transformer, which enables fast and accurate parallel decoding in contrast to the currently prevailing autoregressive models. To the best of our knowledge, this is the first parallel translation system that can be scaled to such a practical scenario like WMT competition. More importantly, our parallel translation system achieves the best BLEU score (35.0) on German->English translation task, outperforming all strong autoregressive counterparts.