RoRA: Efficient Fine-Tuning of LLM with Reliability Optimization for Rank Adaptation

TL;DR

RoRA is a novel method that optimizes the scaling factor in Low-Rank Adaptation, significantly improving fine-tuning performance and robustness of large language models, especially in pruned and uncompressed scenarios.

Contribution

RoRA introduces a new scaling adjustment for LoRA, enhancing performance as rank size increases and outperforming existing methods in accuracy and robustness.

Findings

RoRA outperforms LoRA and DoRA in average accuracy on LLaMA models.

RoRA significantly improves accuracy recovery in pruned models.

Extensive experiments validate RoRA's effectiveness across multiple LLaMA variants.

Abstract

Fine-tuning helps large language models (LLM) recover degraded information and enhance task performance. Although Low-Rank Adaptation (LoRA) is widely used and effective for fine-tuning, we have observed that its scaling factor can limit or even reduce performance as the rank size increases. To address this issue, we propose RoRA (Rank-adaptive Reliability Optimization), a simple yet effective method for optimizing LoRA's scaling factor. By replacing $\alpha/r$ with $\alpha/\sqrt{r}$, RoRA ensures improved performance as rank size increases. Moreover, RoRA enhances low-rank adaptation in fine-tuning uncompressed models and excels in the more challenging task of accuracy recovery when fine-tuning pruned models. Extensive experiments demonstrate the effectiveness of RoRA in fine-tuning both uncompressed and pruned models. RoRA surpasses the state-of-the-art (SOTA) in average accuracy and robustness on LLaMA-7B/13B, LLaMA2-7B, and LLaMA3-8B, specifically outperforming LoRA and DoRA by 6.5% and 2.9% on LLaMA-7B, respectively. In pruned model fine-tuning, RoRA shows significant advantages; for SHEARED-LLAMA-1.3, a LLaMA-7B with 81.4% pruning, RoRA achieves 5.7% higher average accuracy than LoRA and 3.9% higher than DoRA.