Strategic Over-Parameterization for Generalizable Low-Rank Adaptation

TL;DR

LoRA-Over introduces auxiliary parameters during training to enhance the adaptation capacity of low-rank adapters, then collapses them for efficient inference, leading to improved generalization across diverse NLP tasks.

Contribution

This paper presents LoRA-Over, a novel over-parameterization framework that enriches low-rank adapters during training and collapses them at inference, improving PEFT generalization.

Findings

LoRA-Over outperforms vanilla LoRA on multiple NLP benchmarks.

Enriching the optimization landscape enhances adaptation capacity.

Collapse step maintains inference efficiency while improving performance.

Abstract

Adapting large language models (LLMs) to downstream tasks via full fine-tuning is increasingly impractical due to its computational and memory demands. Parameter-efficient fine-tuning (PEFT) approaches such as Low-Rank Adaptation (LoRA) mitigate this by confining updates to a compact set of trainable parameters, but this aggressive reduction often sacrifices generalization, especially under transfer across heterogeneous tasks and domains. We revisit the tension between parameter efficiency and adaptation capacity, and ask whether the two are truly at odds. We answer in the negative by introducing LoRA-Over, a framework grounded in a simple principle: enrich the optimization landscape during training, then collapse the enrichment at inference. LoRA-Over injects auxiliary parameters into the low-rank adapters during training to broaden the effective hypothesis space, and through a decomposition-based reformulation folds them back into a standard low-rank structure with negligible reconstruction error, keeping inference cost identical to vanilla LoRA. Since not all weight matrices benefit equally from added capacity, we further propose two scheduling strategies, one statically predefined and one dynamically determined at runtime, that direct extra capacity where most needed. We evaluate LoRA-Over on language understanding (GLUE, T5-Base), dialogue (MT-Bench), arithmetic reasoning (GSM8K), and code generation (HumanEval), using LLaMA 2-7B and LLaMA 3.1-8B. Across all benchmarks and scales, LoRA-Over consistently outperforms vanilla LoRA, showing that principled over-parameterization designed to vanish at inference is an effective lever for improving PEFT generalization. Code will be released upon acceptance.