Learning When to Adapt

TL;DR

DISeL enhances LoRA by adding input-dependent gates, reducing catastrophic forgetting and providing interpretability during fine-tuning of large models.

Contribution

It introduces DISeL, a lightweight, input-sensitive extension to LoRA that preserves pre-trained behavior while improving task-specific adaptation.

Findings

DISeL reduces forgetting compared to LoRA on multiple tasks.

DISeL maintains competitive fine-tuning accuracy.

Gate activations offer interpretability of adaptation focus.

Abstract

Low-rank adaptation (LoRA) is a widely used parameter-efficient fine-tuning method, yet its learned correction is static: the same low-rank update is applied to every input. This input-agnostic approach creates an inevitable compromise between adapting to the fine-tuning distribution and preserving pre-trained behavior on inputs outside that distribution, contributing to catastrophic forgetting. We introduce DISeL (Dynamic Input-Sensitive LoRA), which augments LoRA modules with lightweight input-dependent gates over individual rank-one components. The gating mechanism is designed to preserve the pre-trained model's behavior by default, while training learns to activate selected components that reduce the fine-tuning loss. DISeL adds only a small number of parameters and preserves the low-rank structure. Across RoBERTa on GLUE, and Llama and Mistral models fine-tuned for mathematical reasoning and code generation, DISeL reduces forgetting relative to LoRA and related variants while maintaining competitive fine-tuning accuracy. In addition, the learned gate activations provide an interpretable diagnostic view of which layers and rank components are most activated during fine-tuning, giving insight into where task-specific adaptation is concentrated. Code available at https://github.com/alizindari/DISeL .