ID-LoRA: Efficient Low-Rank Adaptation Inspired by Matrix Interpolative Decomposition

TL;DR

ID-LoRA introduces a novel parameter-efficient fine-tuning method that extracts and reuses clustered parameter groups from pretrained weights, significantly reducing trainable parameters while maintaining performance across diverse tasks.

Contribution

It proposes a new PEFT framework that leverages clustered parameter groups to form multiple low-rank components sharing a single trainable matrix, improving efficiency and effectiveness.

Findings

Outperforms full fine-tuning and existing PEFT methods on five benchmarks.

Uses up to 46% fewer trainable parameters than standard LoRA.

Surpasses LoRA and variants in multi-task scenarios with only 54% of parameters.

Abstract

LoRA has become a universal Parameter-Efficient Fine-Tuning (PEFT) technique that equips Large Language Models (LLMs) to adapt quickly to new tasks. However, when these models are scaled up, even the latest LoRA variants still introduce considerable overhead in trainable parameters. Conversely, aggressively lowering the rank to curb this overhead markedly degrades performance in complex multi-task settings. We propose ID-LoRA, a novel PEFT framework that breaks the trade-off. Its core innovation lies in extracting and reusing clustered parameter groups from the pretrained weight matrix. These groups are then used to form multiple low-rank components, all of which share only a single initialized trainable low-rank matrix. This approach cuts the number of trainable parameters while keeping the model's capacity intact. We evaluate ID-LoRA on five diverse benchmarks: Mathematical Reasoning, Code Generation, MMLU, CommonsenseQA, and Safety Alignment. ID-LoRA outperforms both full fine-tuning and existing PEFT baselines (e.g., LoRA, DoRA, HydraLoRA) while using up to 46% fewer trainable parameters than the standard LoRA. In multi-task scenarios, it surpasses LoRA and its recent variants (e.g., DoRA and HydraLoRA) on both Code and MMLU tasks, yet requires only 54% of the trainable parameters demanded by the conventional LoRA.