Post-Optimization Adaptive Rank Allocation for LoRA

TL;DR

This paper introduces PARA, a post-optimization method that adaptively allocates ranks in LoRA models using SVD, significantly reducing parameters while maintaining performance.

Contribution

PARA is a novel, data-free, post-hoc compression technique that optimizes LoRA ranks based on spectral importance without retraining.

Findings

PARA reduces LoRA parameters by 75-90%.

PARA preserves original model performance across vision and language tasks.

PARA integrates seamlessly into existing fine-tuning pipelines.

Abstract

Exponential growth in the scale of modern foundation models has led to the widespread adoption of Low-Rank Adaptation (LoRA) as a parameter-efficient fine-tuning technique. However, standard LoRA implementations disregard the varying intrinsic dimensionality of model layers and enforce a uniform rank, leading to parameter redundancy. We propose Post-Optimization Adaptive Rank Allocation (PARA), a data-free compression method for LoRA that integrates seamlessly into existing fine-tuning pipelines. PARA leverages Singular Value Decomposition to prune LoRA ranks using a global threshold over singular values across all layers. This results in non-uniform rank allocation based on layer-wise spectral importance. As a post-hoc method, PARA circumvents the training modifications and resulting instabilities that dynamic architectures typically incur. We empirically demonstrate that PARA reduces parameter count by 75-90\% while preserving the predictive performance of the original, uncompressed LoRA across multiple vision and language benchmarks. Code will be published upon acceptance.