Neural Machine Translation without Embeddings

TL;DR

This paper demonstrates that byte-level neural machine translation without embedding layers can match or outperform traditional subword models, simplifying the architecture and improving performance.

Contribution

It introduces a novel byte-level NMT approach that eliminates the embedding layer, showing competitive results across multiple languages.

Findings

Byte-to-byte translation models outperform subword models in BLEU scores.

Replacing embeddings with one-hot byte representations does not reduce performance.

Decoder-input dropout acts as token dropout, enhancing model robustness.

Abstract

Many NLP models operate over sequences of subword tokens produced by hand-crafted tokenization rules and heuristic subword induction algorithms. A simple universal alternative is to represent every computerized text as a sequence of bytes via UTF-8, obviating the need for an embedding layer since there are fewer token types (256) than dimensions. Surprisingly, replacing the ubiquitous embedding layer with one-hot representations of each byte does not hurt performance; experiments on byte-to-byte machine translation from English to 10 different languages show a consistent improvement in BLEU, rivaling character-level and even standard subword-level models. A deeper investigation reveals that the combination of embeddingless models with decoder-input dropout amounts to token dropout, which benefits byte-to-byte models in particular.