Efficient Inference For Neural Machine Translation

TL;DR

This paper explores combining various optimization techniques to significantly improve inference speed in neural machine translation models while maintaining translation quality, demonstrating up to 109% CPU and 84% GPU speedups.

Contribution

It presents an empirical study on combining known methods like simplified attention, deep encoder-shallow decoder architecture, and pruning to optimize inference speed and reduce parameters.

Findings

Achieves up to 109% CPU speedup and 84% GPU speedup.

Reduces model parameters by 25%.

Maintains BLEU translation quality.

Abstract

Large Transformer models have achieved state-of-the-art results in neural machine translation and have become standard in the field. In this work, we look for the optimal combination of known techniques to optimize inference speed without sacrificing translation quality. We conduct an empirical study that stacks various approaches and demonstrates that combination of replacing decoder self-attention with simplified recurrent units, adopting a deep encoder and a shallow decoder architecture and multi-head attention pruning can achieve up to 109% and 84% speedup on CPU and GPU respectively and reduce the number of parameters by 25% while maintaining the same translation quality in terms of BLEU.