Speeding Up Neural Machine Translation Decoding by Cube Pruning

TL;DR

This paper introduces a cube pruning technique for neural machine translation that significantly accelerates decoding speed by reducing RNN and softmax operations, achieving over three times faster translation without quality loss.

Contribution

The paper adapts cube pruning to neural machine translation, enabling faster decoding by combining similar target states and reducing computational complexity.

Findings

Achieves 3.3x speedup on GPUs

Achieves 3.5x speedup on CPUs

Maintains or improves translation quality

Abstract

Although neural machine translation has achieved promising results, it suffers from slow translation speed. The direct consequence is that a trade-off has to be made between translation quality and speed, thus its performance can not come into full play. We apply cube pruning, a popular technique to speed up dynamic programming, into neural machine translation to speed up the translation. To construct the equivalence class, similar target hidden states are combined, leading to less RNN expansion operations on the target side and less \$\mathrm{softmax}\$ operations over the large target vocabulary. The experiments show that, at the same or even better translation quality, our method can translate faster compared with naive beam search by \$3.3\times\$ on GPUs and \$3.5\times\$ on CPUs.