Challenges in Adapting Multilingual LLMs to Low-Resource Languages using LoRA PEFT Tuning

TL;DR

This paper explores the challenges of adapting multilingual LLMs to low-resource languages like Marathi using LoRA PEFT, revealing mixed evaluation results and emphasizing the need for better assessment methods and native datasets.

Contribution

It investigates the impact of LoRA PEFT on multilingual LLMs for Marathi, highlighting discrepancies between evaluation metrics and manual assessments.

Findings

Fine-tuning improves target language generation.

Evaluation metrics may not reflect true performance.

Reasoning abilities decline after adaptation.

Abstract

Large Language Models (LLMs) have demonstrated remarkable multilingual capabilities, yet challenges persist in adapting these models for low-resource languages. In this study, we investigate the effects of Low-Rank Adaptation (LoRA) Parameter-Efficient Fine-Tuning (PEFT) on multilingual Gemma models for Marathi, a language with limited resources. Using a translated Alpaca dataset with 52,000 instruction-response pairs, our findings reveal that while evaluation metrics often show a performance decline post-fine-tuning, manual assessments frequently suggest that the fine-tuned models outperform their original counterparts. The observations indicate improvements in target language generation capabilities but a reduction in reasoning abilities following language adaptation. These results underscore the need for improved evaluation methodologies and the creation of high-quality native datasets to accurately assess language-specific model performance in low-resource settings.