BiDoRA: Bi-level Optimization-Based Weight-Decomposed Low-Rank Adaptation

TL;DR

BiDoRA introduces a bi-level optimization framework for weight-decomposed low-rank adaptation, improving parameter efficiency and performance in fine-tuning large language models by decoupling magnitude and direction updates.

Contribution

The paper proposes a novel bi-level optimization approach for PEFT that decouples weight decomposition components, reducing overfitting and enhancing adaptation performance.

Findings

BiDoRA achieves a magnitude-direction update correlation of -8.042, closer to full fine-tuning.

Outperforms DoRA and other PEFT methods across diverse NLP tasks.

Statistically significant improvements on the GLUE benchmark with p-value 2.4×10^{-4}.

Abstract

Parameter-efficient fine-tuning (PEFT) is a flexible and efficient method for adapting large language models (LLMs) to downstream tasks. Among these methods, weight-decomposed low-rank adaptation (DoRA) is a promising approach that decomposes weight matrices into magnitude and direction components to mimic full fine-tuning (FT) better. However, DoRA's simultaneous optimization of these components makes it over-expressive, increases the risk of overfitting, and creates a coupled updating pattern that limits its learning capacity. To address these issues, we propose Bi-level Optimization-Based Weight-Decomposed Low-Rank Adaptation (BiDoRA), a novel PEFT method based on a bi-level optimization framework. BiDoRA fundamentally differs from DoRA by optimizing the magnitude and direction in two separate, asynchronous loops using distinct training and validation data splits. This decoupled optimization process effectively mitigates overfitting and allows for more flexible updates that align even more closely with FT. For instance, weight decomposition analysis shows BiDoRA achieves a magnitude-direction update correlation of $-8.042$, significantly closer to the FT ideal compared to $-1.784$ for DoRA. Evaluation of BiDoRA on diverse tasks spanning natural language understanding, generation, token classification, and extremely small biomedical datasets reveals that it consistently outperforms DoRA and a wide range of leading PEFT methods. This improvement is statistically significant, as demonstrated on the GLUE benchmark where BiDoRA surpasses DoRA with a p-value of $2.4\times10^{-4}$ in terms of the Wilcoxon signed-rank test. The code for BiDoRA is available at https://github.com/t2ance/BiDoRA.