Monolingual or Multilingual Instruction Tuning: Which Makes a Better Alpaca

TL;DR

This paper compares monolingual and multilingual instruction tuning of large language models, finding that multilingual approaches are cost-effective and can outperform monolingual tuning in certain scenarios, especially with downsampled data.

Contribution

It provides an empirical analysis of multilingual instruction tuning strategies using Alpaca data, highlighting cost efficiency and robustness advantages.

Findings

Multilingual tuning matches or exceeds monolingual performance.

Downsampled multilingual data can be as effective and more robust.

Multilingual tuning is cost-efficient under fixed computation budgets.

Abstract

Foundational large language models (LLMs) can be instruction-tuned to perform open-domain question answering, facilitating applications like chat assistants. While such efforts are often carried out in a single language, we empirically analyze cost-efficient strategies for multilingual scenarios. Our study employs the Alpaca dataset and machine translations of it to form multilingual data, which is then used to tune LLMs through either low-rank adaptation or full-parameter training. Under a controlled computation budget, comparisons show that multilingual tuning is on par or better than tuning a model for each language. Furthermore, multilingual tuning with downsampled data can be as powerful and more robust. Our findings serve as a guide for expanding language support through instruction tuning.