FlexLoRA: Entropy-Guided Flexible Low-Rank Adaptation

TL;DR

FlexLoRA introduces an entropy-guided, flexible low-rank adaptation method for parameter-efficient fine-tuning of large models, allowing dynamic rank adjustment and improved performance over existing methods.

Contribution

It proposes a novel entropy-based importance measure and a flexible framework for rank pruning and expansion in PEFT, addressing limitations of fixed-rank and heuristic methods.

Findings

Outperforms state-of-the-art PEFT methods across benchmarks.

Effectively prunes and expands ranks based on spectral energy entropy.

Maintains stability with zero-impact initialization for new directions.

Abstract

Large pre-trained models achieve remarkable success across diverse domains, yet fully fine-tuning incurs prohibitive computational and memory costs. Parameter-efficient fine-tuning (PEFT) has thus become a mainstream paradigm. Among them, Low-Rank Adaptation (LoRA) introduces trainable low-rank matrices and shows strong performance, nevertheless, its fixed-rank design limits flexibility. Dynamic rank allocation methods mitigate this issue by pruning redundant directions; however, they often rely on heuristic, element-level metrics that globally sort rank directions without matrix-wise distinction, and they lack mechanisms to expand capacity in layers requiring additional adaptation. To overcome these limitations, we propose FlexLoRA, an entropy-guided flexible low-rank adaptation framework that (i) evaluates matrix importance via spectral energy entropy, (ii) supports rank pruning and expansion under a global budget, and (iii) employs zero-impact initialization for newly added singular directions to ensure stability. By addressing granularity, flexibility, and stability limitations, FlexLoRA provides a more principled solution for PEFT. Extensive experiments show that FlexLoRA consistently outperforms state-of-the-art baselines across benchmarks. Codes are available at https://github.com/Chongjie-Si/Subspace-Tuning.