Word-Level Representation From Bytes For Language Modeling

TL;DR

This paper introduces Byte2Word, a token-free language model that constructs word-level representations directly from bytes using cross-attention, reducing embedding size while maintaining performance.

Contribution

It proposes a novel byte-based word representation method with cross-attention, enabling smaller embeddings and improved robustness, especially for noise and cross-lingual transfer.

Findings

Byte2Word matches BERT's performance on language modeling and text classification.

It uses only 10% of the embedding size compared to traditional models.

The method is robust to synthetic noise and effective in cross-lingual transfer.

Abstract

Modern language models mostly take sub-words as input, a design that balances the trade-off between vocabulary size, number of parameters, and performance. However, sub-word tokenization still has disadvantages like not being robust to noise and difficult to generalize to new languages. Also, the current trend of scaling up models reveals that larger models require larger embeddings but that makes parallelization hard. Previous work on image classification proves splitting raw input into a sequence of chucks is a strong, model-agnostic inductive bias. Based on this observation, we rethink the existing character-aware method that takes character-level inputs but makes word-level sequence modeling and prediction. We overhaul this method by introducing a cross-attention network that builds word-level representation directly from bytes, and a sub-word level prediction based on word-level hidden states to avoid the time and space requirement of word-level prediction. With these two improvements combined, we have a token free model with slim input embeddings for downstream tasks. We name our method Byte2Word and perform evaluations on language modeling and text classification. Experiments show that Byte2Word is on par with the strong sub-word baseline BERT but only takes up 10\% of embedding size. We further test our method on synthetic noise and cross-lingual transfer and find it competitive to baseline methods on both settings.