Dual Decomposition of Weights and Singular Value Low Rank Adaptation

TL;DR

This paper introduces DuDe, a novel weight decomposition method using SVD for Low-rank Adaptation in LLMs, improving training stability and knowledge transfer, with strong empirical results on multiple benchmarks.

Contribution

DuDe is the first approach to decompose weights into magnitude and direction using SVD for PEFT, enhancing stability and transfer in LLM fine-tuning.

Findings

Achieves up to 48.35% accuracy on MMLU

Attains 62.53% accuracy on GSM8K with low variance

Demonstrates improved stability and knowledge transfer

Abstract

Parameter-Efficient Fine-Tuning (PEFT) has emerged as a critical paradigm for adapting Large Language Models (LLMs) to downstream tasks, among which Low-rank Adaptation (LoRA) represents one of the most widely adopted methodologies. However, existing LoRA-based approaches exhibit two fundamental limitations: unstable training dynamics and inefficient knowledge transfer from pre-trained models, both stemming from random initialization of adapter parameters. To overcome these challenges, we propose DuDe, a novel approach that decomposes weight matrices into magnitude and direction components, employing Singular Value Decomposition (SVD) for principled initialization. Our comprehensive evaluation demonstrates DuDe's superior performance and robustness, achieving up to 48.35\% accuracy on MMLU and 62.53\% ($\pm$ 1.59) accuracy on GSM8K. Our theoretical analysis and empirical validation collectively demonstrate that DuDe's decomposition strategy enhances optimization stability and better preserves pre-trained representations, particularly for domain-specific tasks requiring specialized knowledge. The combination of robust empirical performance and rigorous theoretical foundations establishes DuDe as a significant contribution to PEFT methodologies for LLMs.