Neural Machine Translation with Error Correction

TL;DR

This paper proposes an error correction mechanism for neural machine translation that uses a two-stream self-attention approach to reduce error propagation and improve translation accuracy, validated on multiple datasets.

Contribution

Introduces a novel error correction mechanism in NMT using two-stream self-attention from XLNet to enhance translation quality.

Findings

Improves translation accuracy over baseline models.

Effective in reducing error propagation during inference.

Validated on multiple translation datasets.

Abstract

Neural machine translation (NMT) generates the next target token given as input the previous ground truth target tokens during training while the previous generated target tokens during inference, which causes discrepancy between training and inference as well as error propagation, and affects the translation accuracy. In this paper, we introduce an error correction mechanism into NMT, which corrects the error information in the previous generated tokens to better predict the next token. Specifically, we introduce two-stream self-attention from XLNet into NMT decoder, where the query stream is used to predict the next token, and meanwhile the content stream is used to correct the error information from the previous predicted tokens. We leverage scheduled sampling to simulate the prediction errors during training. Experiments on three IWSLT translation datasets and two WMT translation datasets demonstrate that our method achieves improvements over Transformer baseline and scheduled sampling. Further experimental analyses also verify the effectiveness of our proposed error correction mechanism to improve the translation quality.