Token Drop mechanism for Neural Machine Translation

TL;DR

This paper introduces Token Drop, a novel regularization technique for neural machine translation that replaces dropped tokens with a special token and uses self-supervised objectives to enhance model generalization.

Contribution

The paper proposes Token Drop and two self-supervised objectives to improve NMT generalization and reduce overfitting, demonstrating effectiveness on multiple language benchmarks.

Findings

Significant improvements over Transformer baseline.

Enhanced model robustness to unfamiliar inputs.

Effective in Chinese-English and English-Romanian translation tasks.

Abstract

Neural machine translation with millions of parameters is vulnerable to unfamiliar inputs. We propose Token Drop to improve generalization and avoid overfitting for the NMT model. Similar to word dropout, whereas we replace dropped token with a special token instead of setting zero to words. We further introduce two self-supervised objectives: Replaced Token Detection and Dropped Token Prediction. Our method aims to force model generating target translation with less information, in this way the model can learn textual representation better. Experiments on Chinese-English and English-Romanian benchmark demonstrate the effectiveness of our approach and our model achieves significant improvements over a strong Transformer baseline.