Empowering Persian LLMs for Instruction Following: A Novel Dataset and Training Approach

TL;DR

This paper introduces FarsInstruct, a comprehensive dataset and a novel training framework called Co-CoLA, to improve instruction-following capabilities of large language models in Persian, a low-resource language.

Contribution

It presents the first extensive Persian instruction dataset and a multi-task training framework to enhance LLM performance in low-resource language settings.

Findings

FarsInstruct contains 197 templates across 21 datasets.

The combined approach improves Persian LLM instruction-following performance.

Framework enhances multi-task adaptability of LoRA-tuned models.

Abstract

Instruction-tuned large language models have demonstrated remarkable capabilities in following human instructions across various domains. However, their proficiency remains notably deficient in many low-resource languages. To address this challenge, we begin by introducing FarsInstruct a comprehensive instruction dataset designed to enhance the instruction following ability of large language models specifically for the Persian language a significant yet underrepresented language globally. FarsInstruct encompasses a wide range of task types and datasets, each containing a mix of straightforward to complex manual written instructions, as well as translations from the Public Pool of Prompts, ensuring a rich linguistic and cultural representation. Furthermore, we introduce Co-CoLA, a framework designed to enhance the multi-task adaptability of LoRA-tuned models. Through extensive experimental analyses, our study showcases the effectiveness of the FarsInstruct dataset coupled with training by the Co-CoLA framework, in improving the performance of large language models within the Persian context. As of the current writing, FarsInstruct comprises 197 templates across 21 distinct datasets, and we intend to update it consistently, thus augmenting its applicability.