How Vocabulary Sharing Facilitates Multilingualism in LLaMA?

TL;DR

This paper investigates how vocabulary sharing influences the multilingual abilities of LLaMA, revealing that existing models have greater multilingual potential than previously thought and providing guidelines for enhancing performance across languages.

Contribution

It offers an exhaustive analysis of vocabulary sharing effects on LLMs' multilingualism across 101 languages and proposes actionable tuning strategies.

Findings

Existing LLMs have strong multilingual capabilities beyond expectations.

Vocabulary sharing significantly impacts multilingual performance.

Guidelines for language-specific tuning improve multilingual results.

Abstract

Large Language Models (LLMs), often show strong performance on English tasks, while exhibiting limitations on other languages. What is an LLM's multilingual capability when it is trained only on certain languages? The underlying mechanism remains unclear. This study endeavors to examine the multilingual capability of LLMs from the vocabulary sharing perspective by conducting an exhaustive analysis across 101 languages. Through the investigation of the performance gap before and after embedding fine-tuning, we discovered four distinct quadrants. By delving into each quadrant we provide actionable and efficient guidelines for tuning these languages. Extensive experiments reveal that existing LLMs possess multilingual capabilities that surpass our expectations, and we can significantly improve the multilingual performance of LLMs based on these attributes of each quadrant~\footnote{\url{https://github.com/CONE-MT/Vocabulary-Sharing-Facilitates-Multilingualism}.}.