AdaPaD: Adaptive Parallel Deflation for PEFT with Self-Correcting Rank Discovery

TL;DR

AdaPaD introduces a novel method for simultaneously training low-rank components in language models, enabling adaptive, self-correcting rank discovery with proven error decay and competitive empirical results.

Contribution

The paper proposes AdaPaD, a self-correcting, adaptive parallel deflation method for PEFT that trains all rank-1 components simultaneously with dynamic rank discovery.

Findings

Error of each component decays exponentially after warm-up.

AdaPaD achieves competitive performance on GLUE and SQuAD benchmarks.

Adapter size is reduced by an average of 30.7% compared to fixed-rank LoRA.

Abstract

Fine-tuning large language models with LoRA requires choosing a rank r before training starts. Existing approaches either extract rank-1 components sequentially, freezing each component's error permanently into every subsequent residual, or optimize the full low-rank factorization jointly with guarantees that describe only the joint update, not individual rank-1 directions. We present AdaPaD (Adaptive Parallel Deflation), which trains all rank-1 components simultaneously: each worker refines its component against a deflation target built from the latest estimates of all predecessors, and as those estimates improve, the targets improve too. We call this property self-correction: deflation errors converge to zero over rounds rather than persisting as fixed residuals. On top of this backbone, AdaPaD adds advance learning (private pre-training before activation) and per-module dynamic rank discovery (importance-based growth until a shared budget is exhausted), making the rank distribution an output rather than an input. We prove that every component's error decays exponentially after a warm-up period, with a generalization bound that splits into a vanishing algorithmic term and an irreducible statistical floor. Empirically, AdaPaD is competitive with adaptive-rank LoRA baselines on GLUE with DeBERTaV3-base at matched parameter budgets, and competitive with fixed-rank LoRA on Qwen3-0.6B SQuAD/SQuAD v2 while deploying an adapter that is on average 30.7% smaller.