D2-LoRA: A Synergistic Approach to Differential and Directional Low-Rank Adaptation

TL;DR

D2-LoRA is a novel parameter-efficient fine-tuning method that combines low-rank residual updates with architectural innovations, achieving high accuracy across multiple NLP tasks with minimal training data and computational overhead.

Contribution

The paper introduces D2-LoRA, a new low-rank adaptation technique that improves accuracy and stability while maintaining algebraic mergeability and low inference latency.

Findings

Achieves 76.4% accuracy on QA and reading comprehension benchmarks.

Improves average accuracy by 2.2 percentage points over LoRA.

Reduces training volatility by 36% and increases evaluation throughput by 1.91x.

Abstract

We systematically investigate the parameter-efficient fine-tuning design space under practical data and compute constraints, and propose D2-LoRA. D2-LoRA achieves 76.4 percent average accuracy across eight question answering and reading comprehension benchmarks using only 5k training samples per task and two epochs, while preserving algebraic mergeability at inference with near-exact numerical equivalence. The method combines signed low-rank residual updates with additive and subtractive components, together with a train-time column-wise projection that keeps each column close to its original norm. After training, the adapter is merged into a single weight matrix, adding zero inference latency. Compared with LoRA, D2-LoRA improves average accuracy by 2.2 percentage points; at matched parameter counts (LoRA rank 2r versus D2-LoRA rank r), the improvement is 1.6 points, indicating gains from architectural design rather than increased parameterization. Compared with DoRA, it matches or exceeds performance on most tasks. Beyond QA and reading comprehension, D2-LoRA improves generative tasks (plus 1.2 ROUGE-L and plus 1.1 percent win rate) and shows 36 percent lower training volatility. The merge preserves numerical fidelity (mean gap about 0.03 percentage points) and recovers about 1.91x evaluation throughput. Training overhead is 19 percent, comparable to DoRA, and decreases with longer input sequences. We provide a geometric analysis explaining how the projection stabilizes training, together with ablation studies isolating the contribution of each design component.