FIM-LoRA: Task-Informative Rank Allocation for LoRA via Calibration-Time Gradient-Variance Estimation

TL;DR

FIM-LoRA introduces a calibration-time gradient-variance estimation method to allocate ranks adaptively across layers, improving LoRA's efficiency and interpretability without additional training costs.

Contribution

It proposes a lightweight, calibration-based approach to assign layer-specific ranks in LoRA, enhancing performance and interpretability without modifying the training process.

Findings

FIM-LoRA matches standard LoRA performance on GLUE and LLaMA tasks.

The method reduces memory cost by approximately 256x compared to full Fisher estimation.

Layer importance aligns with established transformer layer roles.

Abstract

Low-rank adaptation (LoRA) assigns a uniform rank to every adapted weight matrix - a practical convenience that ignores a fundamental reality: different layers contribute unequally to task adaptation. We address this with a lightweight engineering solution: before fine-tuning begins, run eight calibration backward passes, compute the gradient variance of each LoRA-B matrix as a proxy for layer informativeness, and redistribute the rank budget proportionally. The resulting adapter is a standard LoRA with a per-layer rank pattern - no new parameters, no training overhead, no changes to serving infrastructure. We implement this via an efficient approximation of the empirical Fisher Information Matrix (eFIM) diagonal, restricted to LoRA adapter matrices only, which reduces memory cost by approximately 256x compared to full-model Fisher estimation. On GLUE with DeBERTa-v3-base, FIM-LoRA matches LoRA (88.6 vs. 88.7) at the same parameter budget, and on commonsense reasoning with LLaMA-3-8B reaches 68.5 vs. 68.7 for LoRA. The per-layer rank maps are interpretable: value projections and early-to-middle layers consistently receive higher rank, consistent with established findings on transformer layer roles.