MoCE: Adaptive Mixture of Contextualization Experts for Byte-based Neural Machine Translation

TL;DR

MoCE introduces an adaptive mixture model for byte-based neural machine translation, improving semantic contextualization and outperforming existing methods in multilingual settings without extensive hyper-parameter tuning.

Contribution

The paper proposes MoCE, a novel adaptive mixture of attention heads as contextualization experts, enhancing byte-level translation by dynamically selecting and combining contextualization strategies.

Findings

Outperforms existing byte-based translation methods on Ted-59 dataset.

Requires fewer parameters than subword-based models.

Effectively adapts to language variations without manual hyper-parameter tuning.

Abstract

Byte-based machine translation systems have shown significant potential in massively multilingual settings. Unicode encoding, which maps each character to specific byte(s), eliminates the emergence of unknown words, even in new languages. This avoids out-of-vocabulary risk in multilingual translation and enables broad language scalability. However, byte-level tokenization results in sequences that are hard to interpret due to limited semantic information per byte. Local contextualization has proven effective in assigning initial semantics to tokens, improving sentence comprehension. Nevertheless, variations in encoding rules across languages necessitate an adaptive approach for effective contextualization. To this end, we propose Mixture of Contextualization Experts (MoCE), adaptively selecting and mixing attention heads, which are treated as contextualization experts. This enhances the flexibility of contextualization scales and allows models to search for better contextualization combinations. Experiment results show that our method outperforms existing methods without extensive manual adjustment of hyper-parameters and surpasses subword-based models with fewer parameters in Ted-59 dataset. Our code is available at https://github.com/ictnlp/MoCE.