Integrating Multi-scale Contextualized Information for Byte-based Neural Machine Translation

TL;DR

This paper introduces Multi-Scale Contextualization (MSC), a novel method for byte-based neural machine translation that dynamically integrates multi-scale contextual information, improving translation quality especially in multilingual and out-of-domain settings.

Contribution

The paper proposes MSC, a new approach that learns and dynamically integrates multi-scale contextualized information in byte-based NMT models, addressing limitations of previous tokenization methods.

Findings

MSC outperforms subword-based methods in multilingual translation.

MSC improves translation quality in out-of-domain scenarios.

The method effectively leverages multi-scale context for better translation accuracy.

Abstract

Subword tokenization is a common method for vocabulary building in Neural Machine Translation (NMT) models. However, increasingly complex tasks have revealed its disadvantages. First, a vocabulary cannot be modified once it is learned, making it hard to adapt to new words. Second, in multilingual translation, the imbalance in data volumes across different languages spreads to the vocabulary, exacerbating translations involving low-resource languages. While byte-based tokenization addresses these issues, byte-based models struggle with the low information density inherent in UTF-8 byte sequences. Previous works enhance token semantics through local contextualization but fail to select an appropriate contextualizing scope based on the input. Consequently, we propose the Multi-Scale Contextualization (MSC) method, which learns contextualized information of varying scales across different hidden state dimensions. It then leverages the attention module to dynamically integrate the multi-scale contextualized information. Experiments show that MSC significantly outperforms subword-based and other byte-based methods in both multilingual and out-of-domain scenarios. Code can be found in https://github.com/ictnlp/Multiscale-Contextualization.