ABM-LoRA: Activation Boundary Matching for Fast Convergence in Low-Rank Adaptation

TL;DR

ABM-LoRA introduces an activation boundary alignment technique that significantly speeds up the convergence of low-rank adapters by reducing information loss at initialization, proven effective across multiple architectures and tasks.

Contribution

It presents a novel initialization strategy for low-rank adapters that aligns activation boundaries to improve convergence speed and performance.

Findings

Accelerates convergence across diverse models and tasks.

Achieves state-of-the-art accuracy on VTAB-1K.

Enhances performance on structured reasoning tasks.

Abstract

We propose Activation Boundary Matching for Low-Rank Adaptation (ABM-LoRA), a principled initialization strategy that substantially accelerates the convergence of low-rank adapters. While LoRA offers high parameter efficiency, its random initialization restricts gradient updates to a mismatched tangent space, causing significant information loss and hindering early convergence. Our ABM-LoRA addresses this by aligning the adapter's activation boundaries with those of the pretrained model before downstream training, thereby maximizing the projection of full-parameter gradients into the adapter subspace. This alignment sharply reduces information loss at initialization, yields a lower starting loss, and accelerates convergence. We demonstrate ABM-LoRA's effectiveness across diverse architectures and tasks: language understanding (T5-Base on GLUE), dialogue generation (LLaMA2-7B on WizardLM), and vision recognition (ViT-B/16 on VTAB-1K). On VTAB-1K, it achieves the highest accuracy among all methods, with strong gains on structured reasoning tasks requiring geometric understanding.