MoRE: A Mixture of Low-Rank Experts for Adaptive Multi-Task Learning

TL;DR

This paper introduces MoRE, a mixture of low-rank experts, to improve multi-task fine-tuning of large language models by adaptively selecting low-rank modules, enhancing performance without extra inference cost.

Contribution

The paper proposes a novel MoRE framework that aligns different low-rank experts with tasks and uses an adaptive rank selector, improving multi-task PEFT efficiency and effectiveness.

Findings

MoRE outperforms traditional LoRA in multi-task benchmarks.

MoRE achieves significant performance gains without additional inference cost.

The adaptive rank selector effectively chooses the appropriate expert for each task.

Abstract

With the rapid development of Large Language Models (LLMs), Parameter-Efficient Fine-Tuning (PEFT) methods have gained significant attention, which aims to achieve efficient fine-tuning of LLMs with fewer parameters. As a representative PEFT method, Low-Rank Adaptation (LoRA) introduces low-rank matrices to approximate the incremental tuning parameters and achieves impressive performance over multiple scenarios. After that, plenty of improvements have been proposed for further improvement. However, these methods either focus on single-task scenarios or separately train multiple LoRA modules for multi-task scenarios, limiting the efficiency and effectiveness of LoRA in multi-task scenarios. To better adapt to multi-task fine-tuning, in this paper, we propose a novel Mixture of Low-Rank Experts (MoRE) for multi-task PEFT. Specifically, instead of using an individual LoRA for each task, we align different ranks of LoRA module with different tasks, which we named low-rank experts. Moreover, we design a novel adaptive rank selector to select the appropriate expert for each task. By jointly training low-rank experts, MoRE can enhance the adaptability and efficiency of LoRA in multi-task scenarios. Finally, we conduct extensive experiments over multiple multi-task benchmarks along with different LLMs to verify model performance. Experimental results demonstrate that compared to traditional LoRA and its variants, MoRE significantly improves the performance of LLMs in multi-task scenarios and incurs no additional inference cost. We also release the model and code to facilitate the community.