GateRA: Token-Aware Modulation for Parameter-Efficient Fine-Tuning

TL;DR

GateRA introduces token-aware modulation with adaptive gating and regularization to improve parameter-efficient fine-tuning of large models, focusing updates on challenging tokens and enhancing performance.

Contribution

It proposes a novel token-aware gating framework with entropy regularization for dynamic, selective fine-tuning, advancing beyond static PEFT methods.

Findings

GateRA outperforms prior PEFT methods on reasoning benchmarks.

Adaptive gating effectively suppresses redundant token updates.

Regularization leads to interpretable, sparse adaptation patterns.

Abstract

Parameter-efficient fine-tuning (PEFT) methods, such as LoRA, DoRA, and HiRA, enable lightweight adaptation of large pre-trained models via low-rank updates. However, existing PEFT approaches apply static, input-agnostic updates to all tokens, disregarding the varying importance and difficulty of different inputs. This uniform treatment can lead to overfitting on trivial content or under-adaptation on more informative regions, especially in autoregressive settings with distinct prefill and decoding dynamics. In this paper, we propose GateRA, a unified framework that introduces token-aware modulation to dynamically adjust the strength of PEFT updates. By incorporating adaptive gating into standard PEFT branches, GateRA enables selective, token-level adaptation, preserving pre-trained knowledge for well-modeled inputs while focusing capacity on challenging cases. Empirical visualizations reveal phase-sensitive behaviors, where GateRA automatically suppresses updates for redundant prefill tokens while emphasizing adaptation during decoding. To promote confident and efficient modulation, we further introduce an entropy-based regularization that encourages near-binary gating decisions. This regularization prevents diffuse update patterns and leads to interpretable, sparse adaptation without hard thresholding. Finally, we present a theoretical analysis showing that GateRA induces a soft gradient-masking effect over the PEFT path, enabling continuous and differentiable control over adaptation. Experiments on multiple commonsense reasoning benchmarks demonstrate that GateRA consistently outperforms or matches prior PEFT methods.