LoRA-DA: Data-Aware Initialization for Low-Rank Adaptation via Asymptotic Analysis

TL;DR

This paper introduces LoRA-DA, a data-aware initialization method for Low-Rank Adaptation that leverages target-domain data and asymptotic analysis to improve fine-tuning performance.

Contribution

It develops a theoretical framework and an efficient algorithm for data-aware LoRA initialization, enhancing accuracy, convergence speed, and robustness.

Findings

LoRA-DA outperforms existing initialization methods across multiple benchmarks.

LoRA-DA achieves faster and more stable convergence.

It requires only a small overhead for initialization.

Abstract

LoRA has become a widely adopted method for PEFT, and its initialization methods have attracted increasing attention. However, existing methods have notable limitations: many methods do not incorporate target-domain data, while gradient-based methods exploit data only at a shallow level by relying on one-step gradient decomposition. In this paper, we establish a theoretical framework for data-aware LoRA initialization. Starting from minimizing the expectation of the parameter discrepancy between the fine-tuned and target models, we derive an optimization problem with two components: a bias term, which is related to the parameter distance between the fine-tuned and target models, and is approximated using a Fisher-gradient formulation to preserve anisotropy; and a variance term, which accounts for the uncertainty introduced by sampling stochasticity through the Fisher information. Solving this problem yields an optimal initialization strategy for LoRA, based on which we develop an efficient algorithm, LoRA-DA. Empirical results across multiple benchmarks demonstrate that LoRA-DA consistently improves final accuracy over existing initialization methods. Additional studies show faster, more stable convergence, robustness across ranks, and only a small initialization overhead for LoRA-DA. The source code will be released upon publication.