ConsNoTrainLoRA: Data-driven Weight Initialization of Low-rank Adapters using Constraints

TL;DR

This paper introduces ConsNoTrainLoRA, a data-driven weight initialization method for low-rank adapters that improves convergence and performance in fine-tuning foundation models across various tasks.

Contribution

The paper presents a novel closed-form initialization approach for LoRA weights based on domain shift constraints, eliminating training during initialization and enhancing fine-tuning outcomes.

Findings

Outperforms standard and data-driven initialization methods in experiments.

Enables faster convergence and better performance in image tasks.

Provides a flexible, rank-variable initialization framework.

Abstract

Foundation models are pre-trained on large-scale datasets and subsequently fine-tuned on small-scale datasets using parameter-efficient fine-tuning (PEFT) techniques like low-rank adapters (LoRA). In most previous works, LoRA weight matrices are randomly initialized with a fixed rank across all attachment points. In this paper, we improve convergence and final performance of LoRA fine-tuning, using our proposed data-driven weight initialization method, ConsNoTrainLoRA (CNTLoRA). We express LoRA initialization as a domain shift problem where we use multiple constraints relating the pre-training and fine-tuning activations. By reformulating these constraints, we obtain a closed-form estimate of LoRA weights that depends on pre-training weights and fine-tuning activation vectors and hence requires no training during initialization. This weight estimate is decomposed to initialize the up and down matrices with proposed flexibility of variable ranks. With the proposed initialization method, we fine-tune on downstream tasks such as image generation, image classification and image understanding. Both quantitative and qualitative results demonstrate that CNTLoRA outperforms standard and data-driven weight initialization methods. Extensive analyses and ablations further elucidate the design choices of our framework, providing an optimal recipe for faster convergence and enhanced performance.