Is It Good Data for Multilingual Instruction Tuning or Just Bad Multilingual Evaluation for Large Language Models?

TL;DR

This paper examines how the use of native versus translated data in multilingual instruction tuning and evaluation affects model performance, revealing significant differences and the importance of native resources.

Contribution

It provides a controlled analysis of the impact of native versus translated data on multilingual model tuning and evaluation, highlighting the limitations of translation-based practices.

Findings

Native instruction data improves model performance over translated data.

Translation imperfections can obscure true model capabilities.

Regularization helps bridge performance gaps on structured tasks.

Abstract

Multilingual large language models are designed, claimed, and expected to cater to speakers of varied languages. We hypothesise that the current practices of fine-tuning and evaluating these models may not perfectly align with this objective owing to a heavy reliance on translation, which cannot cover language-specific knowledge but can introduce translation defects. It remains unknown whether the nature of the instruction data has an impact on the model output; conversely, it is questionable whether translated test sets can capture such nuances. Due to the often coupled practices of using translated data in both stages, such imperfections could have been overlooked. This work investigates these issues using controlled native or translated data during the instruction tuning and evaluation stages. We show that native or generation benchmarks reveal a notable difference between native and translated instruction data especially when model performance is high, whereas other types of test sets cannot. The comparison between round-trip and single-pass translations reflects the importance of knowledge from language-native resources. Finally, we demonstrate that regularization is beneficial to bridging this gap on structured but not generative tasks.