Bactrian-X: Multilingual Replicable Instruction-Following Models with Low-Rank Adaptation

TL;DR

Bactrian-X introduces a large multilingual dataset and LoRA-based adapters for instruction tuning, significantly enhancing multilingual model performance and flexibility across 52 languages.

Contribution

The paper presents a new multilingual dataset and a LoRA-based adaptation method, enabling efficient instruction tuning and improved performance for large language models in multiple languages.

Findings

LoRA adapters outperform vanilla models in multilingual tasks

Bactrian-X dataset covers 52 languages with 3.4 million instruction-response pairs

Models trained on Bactrian-X achieve state-of-the-art results in multilingual evaluation

Abstract

Instruction tuning has shown great promise in improving the performance of large language models. However, research on multilingual instruction tuning has been limited due to the scarcity of high-quality instruction-response datasets across different languages. To bridge this gap, we present Bactrian-X, a comprehensive multilingual parallel dataset of 3.4 million instruction-response pairs across 52 languages. Leveraging this dataset, we train a set of adapters using low-rank adaptation (LoRA), which are lightweight components that seamlessly integrate with large language models. These adapters have a substantially lower parameter count than the base model, making them easily replaceable and usable as plug-ins for different languages or language groups. Extensive experiments in various multilingual evaluation settings demonstrate that models derived from LoRA-based training over Bactrian-X outperform both the vanilla models and existing instruction-tuned models. The code and models are publicly available at https://github.com/mbzuai-nlp/bactrian-x