SAMoRA: Semantic-Aware Mixture of LoRA Experts for Task-Adaptive Learning

TL;DR

SAMoRA introduces a semantic-aware routing and task-adaptive scaling framework for multi-task learning with large language models, significantly improving expert specialization and task performance.

Contribution

It proposes a novel semantic-aware router and task-adaptive scaling mechanism for more precise expert routing and contribution regulation in multi-task learning.

Findings

Outperforms state-of-the-art methods on multiple benchmarks.

Demonstrates excellent task generalization capabilities.

Effectively promotes expert specialization and adaptive scaling.

Abstract

The combination of Mixture-of-Experts (MoE) and Low-Rank Adaptation (LoRA) has shown significant potential for enhancing the multi-task learning capabilities of Large Language Models. However, existing methods face two primary challenges: (1)Imprecise Routing in the current MoE-LoRA method fails to explicitly match input semantics with expert capabilities, leading to weak expert specialization. (2)Uniform weight fusion strategies struggle to provide adaptive update strengths, overlooking the varying complexity of different tasks. To address these limitations, we propose SAMoRA (Semantic-Aware Mixture of LoRA Experts), a novel parameter-efficient fine-tuning framework tailored for task-adaptive learning. Specifically, A Semantic-Aware Router is proposed to explicitly align textual semantics with the most suitable experts for precise routing. A Task-Adaptive Scaling mechanism is designed to regulate expert contributions based on specific task requirements dynamically. In addition, a novel regularization objective is proposed to jointly promote expert specialization and effective scaling. Extensive experiments on multiple multi-task benchmarks demonstrate that SAMoRA significantly outperforms the state-of-the-art methods and holds excellent task generalization capabilities. Code is available at https://github.com/boyan-code/SAMoRA