MoR: Mixture of Ranks for Low-Rank Adaptation Tuning

TL;DR

The paper introduces Mixture of Ranks (MoR), a novel low-rank adaptation method that efficiently captures high-rank information and improves multi-task performance without increasing parameters or latency.

Contribution

MoR is a new framework that learns rank-specific information and mathematically transforms low-rank components to approximate high-rank information, enhancing low-rank adaptation.

Findings

MoR improves performance by 1.31% over baselines.

MoR uses only 93.93% of parameters compared to baseline methods.

MoR effectively captures high-rank information through low-rank transformations.

Abstract

Low-Rank Adaptation (LoRA) drives research to align its performance with full fine-tuning. However, significant challenges remain: (1) Simply increasing the rank size of LoRA does not effectively capture high-rank information, which leads to a performance bottleneck.(2) MoE-style LoRA methods substantially increase parameters and inference latency, contradicting the goals of efficient fine-tuning and ease of application. To address these challenges, we introduce Mixture of Ranks (MoR), which learns rank-specific information for different tasks based on input and efficiently integrates multi-rank information. We firstly propose a new framework that equates the integration of multiple LoRAs to expanding the rank of LoRA. Moreover, we hypothesize that low-rank LoRA already captures sufficient intrinsic information, and MoR can derive high-rank information through mathematical transformations of the low-rank components. Thus, MoR can reduces the learning difficulty of LoRA and enhances its multi-task capabilities. MoR achieves impressive results, with MoR delivering a 1.31\% performance improvement while using only 93.93\% of the parameters compared to baseline methods.