CoMoL: Efficient Mixture of LoRA Experts via Dynamic Core Space Merging

TL;DR

CoMoL introduces a dynamic, parameter-efficient MoE-LoRA framework with core space experts and routing, enabling fine-grained adaptation and outperforming existing methods in diverse tasks.

Contribution

The paper proposes CoMoL, a novel MoE-LoRA architecture that enhances expert diversity and parameter efficiency through core space experts and dynamic routing.

Findings

Achieves comparable parameter efficiency to standard LoRA.

Outperforms existing MoE-LoRA methods on multiple tasks.

Maintains high model adaptability with fewer parameters.

Abstract

Large language models (LLMs) achieve remarkable performance on diverse downstream and domain-specific tasks via parameter-efficient fine-tuning (PEFT). However, existing PEFT methods, particularly MoE-LoRA architectures, suffer from limited parameter efficiency and coarse-grained adaptation due to the proliferation of LoRA experts and instance-level routing. To address these issues, we propose Core Space Mixture of LoRA (\textbf{CoMoL}), a novel MoE-LoRA framework that incorporates expert diversity, parameter efficiency, and fine-grained adaptation. Specifically, CoMoL introduces two key components: core space experts and core space routing. Core space experts store each expert in a compact core matrix, preserving diversity while controlling parameter growth. Core space routing dynamically selects and activates the appropriate core experts for each token, enabling fine-grained, input-adaptive routing. Activated core experts are then merged via a soft-merging strategy into a single core expert, which is combined with a shared LoRA to form a specialized LoRA module. Besides, the routing network is projected into the same low-rank space as the LoRA matrices, further reducing parameter overhead without compromising expressiveness. Extensive experiments demonstrate that CoMoL retains the adaptability of MoE-LoRA architectures while achieving parameter efficiency comparable to standard LoRA, consistently outperforming existing methods across multiple tasks.