Revisiting Robust Neural Machine Translation: A Transformer Case Study

TL;DR

This paper investigates the vulnerability of Transformer-based neural machine translation systems to noise and introduces novel training techniques, including Target Augmented Fine-tuning, Controlled Denoising, and Dual-Channel Decoding, to improve noise robustness without inference overhead.

Contribution

The paper presents three new methods to enhance Transformer NMT robustness to noise, focusing on training-phase modifications that do not affect inference.

Findings

Models tolerate up to 10% noise without performance loss

Proposed techniques significantly improve noise robustness in translation

Transformers are more resilient with the new training strategies

Abstract

Transformers (Vaswani et al., 2017) have brought a remarkable improvement in the performance of neural machine translation (NMT) systems but they could be surprisingly vulnerable to noise. In this work, we try to investigate how noise breaks Transformers and if there exist solutions to deal with such issues. There is a large body of work in the NMT literature on analyzing the behavior of conventional models for the problem of noise but Transformers are relatively understudied in this context. Motivated by this, we introduce a novel data-driven technique called Target Augmented Fine-tuning (TAFT) to incorporate noise during training. This idea is comparable to the well-known fine-tuning strategy. Moreover, we propose two other novel extensions to the original Transformer: Controlled Denoising (CD) and Dual-Channel Decoding (DCD), that modify the neural architecture as well as the training process to handle noise. One important characteristic of our techniques is that they only impact the training phase and do not impose any overhead at inference time. We evaluated our techniques to translate the English--German pair in both directions and observed that our models have a higher tolerance to noise. More specifically, they perform with no deterioration where up to 10% of entire test words are infected by noise.