ARD-LoRA: Dynamic Rank Allocation for Parameter-Efficient Fine-Tuning of Foundation Models with Heterogeneous Adaptation Needs

TL;DR

ARD-LoRA introduces a learnable, dynamic rank allocation mechanism for parameter-efficient fine-tuning of foundation models, significantly reducing parameters while maintaining high performance and improving adaptation memory efficiency.

Contribution

It presents a novel framework that automates per-head rank allocation using learnable scaling factors optimized by a meta-objective, advancing parameter-efficient fine-tuning methods.

Findings

Achieves up to 99.3% of full fine-tuning performance with only 0.32% trainable parameters.

Reduces multimodal adaptation memory by 41%.

Outperforms baseline methods like DoRA and AdaLoRA.

Abstract

Conventional Low-Rank Adaptation (LoRA) methods employ a fixed rank, imposing uniform adaptation across transformer layers and attention heads despite their heterogeneous learning dynamics. This paper introduces Adaptive Rank Dynamic LoRA (ARD-LoRA), a novel framework that automates rank allocation through learnable scaling factors. These factors are optimized via a meta-objective balancing task performance and parameter efficiency, incorporating $\ell_1$ sparsity for minimal rank and Total Variation regularization for stable rank transitions. ARD-LoRA enables continuous, differentiable, per-head rank adaptation. Experiments on LLAMA-3.1-70B and PaliGemma-2 demonstrate ARD-LoRA's efficacy, achieving up to 99.3% of full fine-tuning performance with only 0.32% trainable parameters, outperforming strong baselines like DoRA and AdaLoRA. Furthermore, it reduces multimodal adaptation memory by 41%. These results establish dynamic, fine-grained rank allocation as a critical paradigm for efficient foundation model adaptation.