R-LoRA: Randomized Multi-Head LoRA for Efficient Multi-Task Learning

TL;DR

R-LoRA introduces a randomized multi-head approach to enhance multi-task learning in large language models, achieving better performance, reduced memory usage, and faster training compared to traditional LoRA methods.

Contribution

This paper proposes R-LoRA, a novel multi-head randomization technique that improves multi-task learning efficiency and effectiveness in large language models.

Findings

R-LoRA outperforms standard LoRA in multi-task benchmarks.

It reduces GPU memory consumption and training time.

Increased head diversity improves task-specific feature learning.

Abstract

Fine-tuning large language models (LLMs) is computationally expensive, and Low-Rank Adaptation (LoRA) provides a cost-effective solution by approximating weight updates through low-rank matrices. In real-world scenarios, LLMs are fine-tuned on data from multiple domains to perform tasks across various fields, embodying multi-task learning (MTL). LoRA often underperforms in such complex scenarios. To enhance LoRA's capability in multi-task learning, we propose R-LoRA, which incorporates Multi-Head Randomization. Multi-Head Randomization diversifies the head matrices through Multi-Head Dropout and Multi-Head Random Initialization, enabling more efficient learning of task-specific features while maintaining shared knowledge representation. Our approach not only improves performance in MTL but also reduces GPU memory usage and training time. Experiments show that R-LoRA's gains stem from increased diversity in the head matrices, demonstrating its effectiveness for multi-task learning. The code is available at https://github.com/jinda-liu/R-LoRA