HRP: High-Rank Preheating for Superior LoRA Initialization

TL;DR

This paper introduces High-Rank Preheating (HRP), a novel initialization method for LoRA that improves fine-tuning results by approximating the optimal initial direction through a pretraining phase, backed by theoretical and experimental validation.

Contribution

The paper proposes HRP, a new initialization technique for LoRA that enhances fine-tuning performance by using a high-rank pretraining step to better approximate the ideal initial direction.

Findings

HRP significantly improves LoRA's fine-tuning results.

Theoretical analysis shows HRP's effectiveness increases with preheating rank.

Extensive experiments demonstrate HRP outperforms existing initialization methods.

Abstract

This paper studies the crucial impact of initialization in Low-Rank Adaptation (LoRA). Through theoretical analysis, we demonstrate that the fine-tuned result of LoRA is highly sensitive to initialization, which is likely to lead suboptimal low-rank results. While this issue can be mitigated by adjusting the initial direction towards the main singular vectors of the target $\Delta W$, which is, however, typically unknown in real-world scenarios. To approximate this initial direction, we propose High-Rank Preheating (HRP), which first trains LoRA with a higher preheating rank for a few steps, then uses the main singular vectors of the derived $BA^\top$ as initialization for the main fine-tuning process. With only a modification in the initial direction, we prove that HRP makes LoRA achieve better fine-tuned results than random initialization in expectation, and the enhancement grows with the preheating rank. We validate our theoretical findings through extensive experiments in various models and tasks, where HRP significantly enhances LoRA's effectiveness and outperforms other initialization strategies and other LoRA variants.