Mixture-of-LoRAs: An Efficient Multitask Tuning for Large Language Models

TL;DR

This paper introduces Mixture-of-LoRAs, a parameter-efficient multi-task tuning method for LLMs that combines domain-specific modules with explicit routing, improving task performance and adaptability.

Contribution

It proposes a novel multi-task tuning architecture that combines multiple LoRA modules with routing strategies, enabling efficient domain adaptation and reducing task interference.

Findings

Superior performance on diverse tasks

Robust multi-task learning capabilities

Effective domain-specific adaptation

Abstract

Instruction Tuning has the potential to stimulate or enhance specific capabilities of large language models (LLMs). However, achieving the right balance of data is crucial to prevent catastrophic forgetting and interference between tasks. To address these limitations and enhance training flexibility, we propose the Mixture-of-LoRAs (MoA) architecture which is a novel and parameter-efficient tuning method designed for multi-task learning with LLMs. In this paper, we start by individually training multiple domain-specific LoRA modules using corresponding supervised corpus data. These LoRA modules can be aligned with the expert design principles observed in Mixture-of-Experts (MoE). Subsequently, we combine the multiple LoRAs using an explicit routing strategy and introduce domain labels to facilitate multi-task learning, which help prevent interference between tasks and ultimately enhances the performance of each individual task. Furthermore, each LoRA model can be iteratively adapted to a new domain, allowing for quick domain-specific adaptation. Experiments on diverse tasks demonstrate superior and robust performance, which can further promote the wide application of domain-specific LLMs.