An Empirical Comparison of Vocabulary Expansion and Initialization Approaches for Language Models

TL;DR

This paper compares different vocabulary expansion and initialization methods for multilingual language models, introducing a simple yet effective approach called CW2V that performs on par with more complex techniques across multiple languages and tasks.

Contribution

It provides a theoretical foundation for initialization within the convex hull and introduces CW2V, a novel method that does not rely on cross-lingual embeddings, with extensive empirical evaluation.

Findings

CW2V performs as well or better than advanced methods.

Simple multivariate initialization matches complex approaches.

Efficient multilingual model adaptation is possible with simpler methods.

Abstract

Language Models (LMs) excel in natural language processing tasks for English but show reduced performance in most other languages. This problem is commonly tackled by continually pre-training and fine-tuning these models for said languages. A significant issue in this process is the limited vocabulary coverage in the original model's tokenizer, leading to inadequate representation of new languages and necessitating an expansion of the tokenizer. The initialization of the embeddings corresponding to new vocabulary items presents a further challenge. Current strategies require cross-lingual embeddings and lack a solid theoretical foundation as well as comparisons with strong baselines. In this paper, we first establish theoretically that initializing within the convex hull of existing embeddings is a good initialization, followed by a novel but simple approach, Constrained Word2Vec (CW2V), which does not require cross-lingual embeddings. Our study evaluates different initialization methods for expanding RoBERTa and LLaMA 2 across four languages and five tasks. The results show that CW2V performs equally well or even better than more advanced techniques. Additionally, simpler approaches like multivariate initialization perform on par with these advanced methods indicating that efficient large-scale multilingual continued pretraining can be achieved even with simpler initialization methods. We release our code publicly (https://github.com/AI4Bharat/VocabAdaptation_LLM/tree/CW2V).