OSoRA: Output-Dimension and Singular-Value Initialized Low-Rank Adaptation

TL;DR

OSoRA introduces a novel low-rank adaptation method for fine-tuning large language models by combining SVD with learnable scaling, significantly reducing computational costs while maintaining or improving performance.

Contribution

It extends Low-Rank Adaptation (LoRA) by integrating SVD and learnable scaling vectors, enabling efficient fine-tuning with fewer resources and comparable or better results.

Findings

Achieves comparable or superior performance to LoRA and VeRA.

Reduces computational resource requirements during fine-tuning.

Maintains linear parameter scaling with increasing rank.

Abstract

Fine-tuning Large Language Models (LLMs) has become increasingly challenging due to their massive scale and associated computational costs. Parameter-Efficient Fine-Tuning (PEFT) methodologies have been proposed as computational alternatives; however, their implementations still require significant resources. In this paper, we present OSoRA (Output-Dimension and Singular-Value Initialized Low-Rank Adaptation), a novel PEFT method for LLMs. OSoRA extends Low-Rank Adaptation (LoRA) by integrating Singular Value Decomposition (SVD) with learnable scaling vectors in a unified framework. It first performs an SVD of pre-trained weight matrices, then optimizes an output-dimension vector during training, while keeping the corresponding singular vector matrices frozen. OSoRA substantially reduces computational resource requirements by minimizing the number of trainable parameters during fine-tuning. Comprehensive evaluations across mathematical reasoning, common sense reasoning, and other benchmarks demonstrate that OSoRA achieves comparable or superior performance to state-of-the-art methods like LoRA and VeRA, while maintaining a linear parameter scaling even as the rank increases to higher dimensions. Our ablation studies further confirm that jointly training both the singular values and the output-dimension vector is critical for optimal performance.