Rethinking Adapter Placement: A Dominant Adaptation Module Perspective

TL;DR

This paper introduces PAGE, a gradient sensitivity probe revealing a dominant adaptation module in models, leading to a new placement method, DomLoRA, that outperforms traditional LoRA with fewer parameters.

Contribution

The paper identifies a single dominant adaptation module for LoRA placement using PAGE, and proposes DomLoRA, a minimal yet effective fine-tuning approach.

Findings

PAGE concentrates on a single shallow FFN down-projection.

DomLoRA with ~0.7% parameters outperforms vanilla LoRA.

DomLoRA improves various LoRA variants across tasks.

Abstract

Low-rank adaptation (LoRA) is a widely used parameter-efficient fine-tuning method that places trainable low-rank adapters into frozen pre-trained models. Recent studies show that using fewer LoRA adapters may still maintain or even improve performance, but existing methods still distribute adapters broadly, leaving where to place a limited number of adapters to maximize performance largely open. To investigate this, we introduce PAGE (Projected Adapter Gradient Energy), a gradient-based sensitivity probe that estimates the initial trainable gradient energy available to each candidate LoRA adapter. Surprisingly, we find that PAGE is highly concentrated on a single shallow FFN down-projection across two model families and four downstream tasks. We term this module the dominant adaptation module and show that its layer index is architecture-dependent but task-stable. Motivated by this finding, we propose DomLoRA, a placement method that places a single adapter at the dominant adaptation module. With only ~0.7% of vanilla LoRA's trainable parameters, DomLoRA outperforms it on average across various downstream tasks, including instruction following, mathematical reasoning, code generation, and multi-turn conversation. This method also improves other LoRA variants, supporting the dominant adaptation module perspective as a practical placement guideline.