Turning English-centric LLMs Into Polyglots: How Much Multilinguality Is Needed?

TL;DR

This study explores the minimal multilingual training needed to enable English-centric large language models to generalize across multiple languages, showing that training on just a few languages can be effective.

Contribution

It demonstrates that multilingual instruction tuning with only two to three languages suffices for cross-lingual generalization in English-centric LLMs, highlighting the importance of pretraining exposure.

Findings

Multilingual instruction tuning improves cross-lingual transfer.

Few languages (2-3) are enough for effective generalization.

Most beneficial for generative, input/output language agreement tasks.

Abstract

The vast majority of today's large language models (LLMs) are English-centric, having been pretrained predominantly on English text. Yet, in order to meet user expectations, models need to be able to respond appropriately in multiple languages once deployed in downstream applications. This requires strong cross-lingual transfer abilities. In this work, we investigate the minimal amount of multilinguality required during finetuning to elicit cross-lingual generalisation in English-centric LLMs. In experiments across four LLMs, we find that multilingual instruction tuning with as few as two to three languages is both necessary and sufficient to elicit effective cross-lingual generalisation, with the limiting factor being the degree to which a target language is seen during pretraining. Evaluations on five different tasks further reveal that multilingual instruction tuning is most beneficial for generative tasks that assume input/output language agreement, such as in chat settings, while being of less importance for highly structured classification-style tasks. Our code and data is available at https://github.com/ZurichNLP/multilingual-instruction-tuning.