The RoyalFlush System for the WMT 2022 Efficiency Task

TL;DR

The paper introduces the RoyalFlush system for WMT 2022, employing a hybrid two-stage translation method that balances speed and quality, achieving significant inference speed improvements over previous systems.

Contribution

It proposes Hybrid Regression Translation (HRT), a novel two-stage translation approach combining autoregressive and non-autoregressive methods, with techniques to optimize speed and performance.

Findings

Achieves 80% faster inference speed

Maintains translation quality comparable to autoregressive models

Reaches over 6,000 words per second on GPU

Abstract

This paper describes the submission of the RoyalFlush neural machine translation system for the WMT 2022 translation efficiency task. Unlike the commonly used autoregressive translation system, we adopted a two-stage translation paradigm called Hybrid Regression Translation (HRT) to combine the advantages of autoregressive and non-autoregressive translation. Specifically, HRT first autoregressively generates a discontinuous sequence (e.g., make a prediction every $k$ tokens, $k>1$) and then fills in all previously skipped tokens at once in a non-autoregressive manner. Thus, we can easily trade off the translation quality and speed by adjusting $k$. In addition, by integrating other modeling techniques (e.g., sequence-level knowledge distillation and deep-encoder-shallow-decoder layer allocation strategy) and a mass of engineering efforts, HRT improves 80\% inference speed and achieves equivalent translation performance with the same-capacity AT counterpart. Our fastest system reaches 6k+ words/second on the GPU latency setting, estimated to be about 3.1x faster than the last year's winner.