Not All Directions Matter: Towards Structured and Task-Aware Low-Rank Model Adaptation

TL;DR

StructLoRA enhances low-rank model adaptation by filtering task-irrelevant directions and enforcing inter-layer consistency, leading to state-of-the-art results without increasing inference costs.

Contribution

It introduces a dual-component framework that addresses semantic drift and structural incoherence in LoRA, improving performance across various models and data regimes.

Findings

Outperforms vanilla LoRA and other methods on multiple models.

Significant gains in low-rank and low-data scenarios.

Operates only during training, incurring no inference overhead.

Abstract

Low-Rank Adaptation (LoRA) has become a cornerstone of parameter-efficient fine-tuning (PEFT). Yet, its efficacy is hampered by two fundamental limitations: semantic drift, by treating all update directions with equal importance, and structural incoherence, from adapting layers independently, resulting in suboptimal, uncoordinated updates. To remedy these, we propose StructLoRA, a framework that addresses both limitations through a principled, dual-component design: (1) an Information Bottleneck-guided filter that prunes task-irrelevant directions to mitigate semantic drift, and (2) a lightweight, training-only graph-based coordinator that enforces inter-layer consistency to resolve structural incoherence. Extensive experiments across large language model , vision language model, and vision model (including LLaMA, LLaVA, and ViT) demonstrate that StructLoRA consistently establishes a new state-of-the-art, outperforming not only vanilla LoRA but also advanced dynamic rank allocation and sparsity-based methods. Notably, the benefits are particularly pronounced in challenging low-rank and low-data regimes. Crucially, since our proposed modules operate only during training, StructLoRA enhances performance with zero additional inference cost, advancing the focus of PEFT -- from mere parameter compression to a more holistic optimization of information quality and structural integrity.