Rethinking Adapter Placement: A Dominant Adaptation Module Perspective
Suoxin Zhang, Run He, Di Fang, Xiang Tan, Kaixuan Chen, Huiping Zhuang

TL;DR
This paper introduces PAGE, a gradient sensitivity probe revealing a dominant adaptation module in models, leading to a new placement method, DomLoRA, that outperforms traditional LoRA with fewer parameters.
Contribution
The paper identifies a single dominant adaptation module for LoRA placement using PAGE, and proposes DomLoRA, a minimal yet effective fine-tuning approach.
Findings
PAGE concentrates on a single shallow FFN down-projection.
DomLoRA with ~0.7% parameters outperforms vanilla LoRA.
DomLoRA improves various LoRA variants across tasks.
Abstract
Low-rank adaptation (LoRA) is a widely used parameter-efficient fine-tuning method that places trainable low-rank adapters into frozen pre-trained models. Recent studies show that using fewer LoRA adapters may still maintain or even improve performance, but existing methods still distribute adapters broadly, leaving where to place a limited number of adapters to maximize performance largely open. To investigate this, we introduce PAGE (Projected Adapter Gradient Energy), a gradient-based sensitivity probe that estimates the initial trainable gradient energy available to each candidate LoRA adapter. Surprisingly, we find that PAGE is highly concentrated on a single shallow FFN down-projection across two model families and four downstream tasks. We term this module the dominant adaptation module and show that its layer index is architecture-dependent but task-stable. Motivated by this…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
