Adaptive LoRA Merge with Parameter Pruning for Low-Resource Generation

TL;DR

This paper introduces an adaptive LoRA merge technique with parameter pruning that enhances low-resource language generation by fine-tuning and updating LoRA modules, leading to improved task adaptability across multiple languages and domains.

Contribution

It presents a novel LoRA merge method that updates and prunes parameters during fine-tuning, improving adaptability for low-resource language tasks.

Findings

Significant improvements in summarization performance across multiple languages.

Enhanced task adaptability over previous LoRA merge methods.

Effective parameter pruning contributes to better low-resource learning.

Abstract

This study proposes a simple yet effective LoRA merge method to achieve LLM adaptation for low-resource language generation tasks. The LoRA merge technique, which integrates multiple LoRA modules trained on different tasks, has gained attention as an effective and efficient approach for adapting LLMs to target tasks. However, previous methods are limited in adaptability as they keep the LoRA parameters frozen. Additionally, the low-resource problem has been out of their scope. We propose a LoRA merge method that updates and prunes LoRA parameters through fine-tuning with minimal target task data, which allows finer-grained adjustments of LoRA parameters and enhancement of task adaptability. Extensive experiments have been conducted taking summarization as a benchmark task. Our datasets cover various domains and multiple languages of English and Japanese. The results confirm that the proposed method achieves significant and consistent improvements in task adaptability over the previous methods.