Cross-Lingual Optimization for Language Transfer in Large Language Models

TL;DR

This paper introduces Cross-Lingual Optimization (CLO), a method that efficiently transfers large language models to new languages while maintaining English capabilities, especially effective in low-resource settings.

Contribution

CLO leverages English data and translation models to improve multilingual transfer, outperforming standard fine-tuning in low-resource scenarios with less data.

Findings

CLO outperforms SFT in target language proficiency and English retention.

CLO with fewer samples surpasses SFT with more data in low-resource languages.

CLO remains robust across varying data quantities, unlike SFT.

Abstract

Adapting large language models to other languages typically employs supervised fine-tuning (SFT) as a standard approach. However, it often suffers from an overemphasis on English performance, a phenomenon that is especially pronounced in data-constrained environments. To overcome these challenges, we propose \textbf{Cross-Lingual Optimization (CLO)} that efficiently transfers an English-centric LLM to a target language while preserving its English capabilities. CLO utilizes publicly available English SFT data and a translation model to enable cross-lingual transfer. We conduct experiments using five models on six languages, each possessing varying levels of resource. Our results show that CLO consistently outperforms SFT in both acquiring target language proficiency and maintaining English performance. Remarkably, in low-resource languages, CLO with only 3,200 samples surpasses SFT with 6,400 samples, demonstrating that CLO can achieve better performance with less data. Furthermore, we find that SFT is particularly sensitive to data quantity in medium and low-resource languages, whereas CLO remains robust. Our comprehensive analysis emphasizes the limitations of SFT and incorporates additional training strategies in CLO to enhance efficiency.