Learning in the Fisher Subspace: A Guided Initialization for LoRA Fine-Tuning

TL;DR

This paper introduces a Fisher-guided, data-aware initialization method for LoRA fine-tuning of large language models, improving adaptation by selecting more relevant subspaces based on downstream data sensitivity.

Contribution

It proposes a novel Fisher-guided framework that leverages curvature information to better align LoRA subspace selection with task-specific data, outperforming existing methods.

Findings

Data-aware initialization improves downstream performance.

Fisher-guided approach effectively captures task relevance.

Method demonstrates consistent gains across diverse tasks.

Abstract

LoRA adapts large language models (LLMs) by restricting updates to low-rank subspaces of pre-trained weights. While this substantially reduces training cost, the effectiveness of adaptation critically depends on which subspace is chosen at initialization: a poor initialization that allocates capacity to task-irrelevant directions can severely hinder downstream performance. Existing initialization strategies primarily rely on the intrinsic properties of pre-trained weights, implicitly assuming that weight geometry alone reflects task relevance. However, such criteria overlook how the model interacts with the downstream data distribution. In this work, we formulate LoRA initialization as identifying the degree of impact of directions in parameter space under the target data distribution. We argue that data-aware sensitivity, rather than weight-only magnitude, should govern the choice of adaptation subspaces. Building on this perspective, we propose a Fisher-guided framework that leverages curvature information induced by downstream data to characterize how parameter perturbations influence model predictions. This perspective yields a principled, task-dependent criterion for selecting LoRA directions that better align adaptation with the target objective. Empirical results across diverse tasks and modalities demonstrate that data-aware initialization consistently and significantly improves downstream performance over existing approaches.