Each Rank Could be an Expert: Single-Ranked Mixture of Experts LoRA for Multi-Task Learning

TL;DR

This paper introduces SMoRA, a novel multi-task learning method that treats each LoRA rank as an expert, enabling better knowledge sharing and task performance with fewer parameters.

Contribution

It unifies single LoRA and multi-LoRA MoE into a single framework and proposes SMoRA, which improves multi-task learning by dynamic rank-wise activation.

Findings

Finer-grained LoRA partitioning improves performance across tasks.

SMoRA activates fewer parameters but achieves better results.

Dynamic rank-wise activation enhances knowledge sharing.

Abstract

Low-Rank Adaptation (LoRA) is widely used for adapting large language models (LLMs) to specific domains due to its efficiency and modularity. Meanwhile, vanilla LoRA struggles with task conflicts in multi-task scenarios. Recent works adopt Mixture of Experts (MoE) by treating each LoRA module as an expert, thereby mitigating task interference through multiple specialized LoRA modules. While effective, these methods often isolate knowledge within individual tasks, failing to fully exploit the shared knowledge across related tasks. In this paper, we establish a connection between single LoRA and multi-LoRA MoE, integrating them into a unified framework. We demonstrate that the dynamic routing of multiple LoRAs is functionally equivalent to rank partitioning and block-level activation within a single LoRA. We further empirically demonstrate that finer-grained LoRA partitioning, within the same total and activated parameter constraints, leads to better performance gains across heterogeneous tasks. Building on these findings, we propose Single-ranked Mixture of Experts LoRA (\textbf{SMoRA}), which embeds MoE into LoRA by \textit{treating each rank as an independent expert}. With a \textit{dynamic rank-wise activation} mechanism, SMoRA promotes finer-grained knowledge sharing while mitigating task conflicts. Experiments demonstrate that SMoRA activates fewer parameters yet achieves better performance in multi-task scenarios.