GAIN: Multiplicative Modulation for Domain Adaptation

TL;DR

This paper introduces GAIN, a multiplicative method for domain adaptation of large language models that reduces forgetting and improves performance without additional data or regularization.

Contribution

GAIN is a simple multiplicative approach that preserves the pretrained weight space, outperforming LoRA and EWC in reducing forgetting during domain adaptation.

Findings

GAIN improves earlier-domain perplexity by 7-13%.

GAIN matches replay-augmented LoRA without storing prior data.

GAIN dominates EWC on the forgetting-adaptation Pareto front.

Abstract

Adapting LLMs to new domains causes forgetting because standard methods (e.g., full fine-tuning, LoRA) inject new directions into the weight space. We show that forgetting is governed by one algebraic property: whether the update preserves the column span of the pretrained weight matrix (Proposition 1). We propose GAIN, the simplest multiplicative alternative (W_new = S * W), which satisfies this by construction and can be absorbed into existing weights for zero inference cost. Across five models (774M to 70B) adapted sequentially over eight domains, GAIN improves earlier-domain perplexity by 7-13%, while LoRA degrades it by 18-36%. GAIN matches replay-augmented LoRA without storing prior data and dominates EWC on the forgetting-adaptation Pareto front. While LoRA can only reduce forgetting by sacrificing in-domain adaptation, GAIN achieves both with no domain boundaries and no regularization. The principle generalises: (IA)^3, an independent multiplicative method, also improves earlier domains.