Efficient and Effective Vocabulary Expansion Towards Multilingual Large Language Models

TL;DR

This paper presents EEVE-Korean-v1.0, a multilingual large language model with an efficient vocabulary expansion method that significantly improves non-English language understanding with less training data, outperforming many existing models.

Contribution

The paper introduces EEVE-Vocabulary Expansion (EEVE), a novel method enabling rapid non-English proficiency in large language models through parameter freezing and subword initialization.

Findings

EEVE method boosts non-English performance with only 2 billion tokens

EEVE-Korean-10.8B-v1.0 outperforms other Korean models on Open Ko-LLM leaderboard

Models are open-sourced on Hugging Face for community use

Abstract

This report introduces \texttt{EEVE-Korean-v1.0}, a Korean adaptation of large language models that exhibit remarkable capabilities across English and Korean text understanding. Building on recent highly capable but English-centric LLMs, such as SOLAR-10.7B and Phi-2, where non-English texts are inefficiently processed with English-centric tokenizers, we present an efficient and effective vocabulary expansion (EEVE) method, which encompasses parameter freezing and subword initialization. In contrast to previous efforts that believe new embeddings require trillions of training tokens, we show that our method can significantly boost non-English proficiency within just 2 billion tokens. Surpassing most instruction-tuned LLMs on the Open Ko-LLM Leaderboard, as of January 2024, our model \texttt{EEVE-Korean-10.8B-v1.0} ranks as the leading Korean pre-trained model in the open-source community, according to Hugging Face's leaderboard. We open-source our models on Huggingface to empower the open research community in various languages.