Synthetic and Natural Noise Both Break Neural Machine Translation

TL;DR

This paper demonstrates that neural machine translation models are highly sensitive to both synthetic and natural noise, and proposes a character convolutional neural network approach to improve robustness against such noise.

Contribution

The paper introduces a character convolutional neural network model that enhances NMT robustness to various noise types, addressing a key brittleness issue.

Findings

State-of-the-art NMT models fail on noisy texts

Robust training improves translation of noisy data

Character CNNs learn noise-invariant representations

Abstract

Character-based neural machine translation (NMT) models alleviate out-of-vocabulary issues, learn morphology, and move us closer to completely end-to-end translation systems. Unfortunately, they are also very brittle and easily falter when presented with noisy data. In this paper, we confront NMT models with synthetic and natural sources of noise. We find that state-of-the-art models fail to translate even moderately noisy texts that humans have no trouble comprehending. We explore two approaches to increase model robustness: structure-invariant word representations and robust training on noisy texts. We find that a model based on a character convolutional neural network is able to simultaneously learn representations robust to multiple kinds of noise.