GoRA: Gradient-driven Adaptive Low Rank Adaptation

TL;DR

GoRA introduces a gradient-driven adaptive framework for low-rank adaptation of large language models, dynamically optimizing rank and initialization to improve performance and efficiency over existing methods.

Contribution

It is the first to unify adaptive rank selection and initialization in a single framework using gradient information during training.

Findings

Outperforms existing LoRA methods across various architectures.

Achieves significant performance gains, e.g., 5.13-point improvement on Llama3.1-8B-Base.

Outperforms full fine-tuning in high-rank settings.

Abstract

Low-Rank Adaptation (LoRA) is a crucial method for efficiently fine-tuning large language models (LLMs), with its effectiveness influenced by two key factors: rank selection and weight initialization. While numerous LoRA variants have been proposed to improve performance by addressing one of these aspects, they often compromise usability or computational efficiency. In this paper, we analyze and identify the core limitations of existing approaches and propose a novel framework--GoRA (Gradient-driven Adaptive Low Rank Adaptation)--that simultaneously adapts both the rank and initialization strategy within a unified framework. GoRA leverages gradient information during training to dynamically assign optimal ranks and initialize low-rank adapter weights in an adaptive manner. To our knowledge, GoRA is the first method that not only addresses the limitations of prior approaches--which often focus on either rank selection or initialization in isolation--but also unifies both aspects within a single framework, enabling more effective and efficient adaptation. Extensive experiments across various architectures and modalities show that GoRA consistently outperforms existing LoRA-based methods while preserving the efficiency of vanilla LoRA. For example, when fine-tuning Llama3.1-8B-Base for mathematical reasoning, GoRA achieves a 5.13-point improvement over standard LoRA and even outperforms full fine-tuning by 2.05 points under high-rank settings. Code is available at: https://github.com/hhnqqq/MyTransformers.