Just Go Parallel: Improving the Multilingual Capabilities of Large Language Models

TL;DR

This paper systematically investigates how incorporating parallel data can enhance the multilingual translation and reasoning abilities of large language models, challenging the notion that scale alone suffices.

Contribution

It provides a controlled experimental analysis showing that adding parallel data significantly boosts LLMs' multilingual performance, especially in translation and reasoning tasks.

Findings

Parallel data improves translation accuracy.

Multilingual reasoning capabilities are enhanced with parallel data.

Scale alone does not fully account for multilingual abilities.

Abstract

Large language models (LLMs) have demonstrated impressive translation capabilities even without being explicitly trained on parallel data. This remarkable property has led some to believe that parallel data is no longer necessary for building multilingual language models. While some attribute this to the emergent abilities of LLMs due to scale, recent work suggests that it is actually caused by incidental bilingual signals present in the training data. Various methods have been proposed to maximize the utility of parallel data to enhance the multilingual capabilities of multilingual encoder-based and encoder-decoder language models. However, some decoder-based LLMs opt to ignore parallel data instead. In this work, we conduct a systematic study on the impact of adding parallel data on LLMs' multilingual capabilities, focusing specifically on translation and multilingual common-sense reasoning. Through controlled experiments, we demonstrate that parallel data can significantly improve LLMs' multilingual capabilities.