HiP-LoRA: Budgeted Spectral Plasticity for Robust Low-Rank Adaptation

TL;DR

HiP-LoRA introduces a spectrum-aware adaptation method for foundation models that reduces interference and preserves capabilities during low-rank fine-tuning by decomposing updates into principal and residual channels.

Contribution

The paper presents HiP-LoRA, a novel spectral adaptation framework that leverages cached SVD to improve robustness and reduce interference in low-rank model fine-tuning.

Findings

HiP-LoRA significantly reduces pretraining degradation.

It outperforms baselines in interference-sensitive tasks.

It improves multi-adapter merging stability.

Abstract

Adapting foundation models under resource budgets relies heavily on Parameter-Efficient Fine-Tuning (PEFT), with LoRA being a standard modular solution. However, LoRA suffers from spectral interference. Low-rank updates often concentrate energy on the leading singular directions of pretrained weights, perturbing general capabilities and causing catastrophic forgetting and fragile multi-adapter merging. To resolve this, we propose HiP-LoRA, a spectrum-aware adaptation framework. Utilizing the cached singular value decomposition (SVD) of pretrained layers, HiP-LoRA decomposes updates into two channels: a principal channel within the dominant singular subspace, and a residual low-rank channel in the orthogonal complement. A singular-value-weighted stability budget on the principal channel continuously balances pretrained behavior preservation with task-specific plasticity. Experiments on Llama-3.1-8B demonstrate that under matched budgets, HiP-LoRA drastically reduces pretraining degradation and multi-adapter MergeFail, robustly outperforming baselines in interference-sensitive tasks like continual tuning and knowledge editing.