Continual Gradient Low-Rank Projection Fine-Tuning for LLMs

TL;DR

The paper introduces GORP, a novel continual learning method for LLMs that combines full and low-rank parameters to improve learning efficiency and knowledge transfer while reducing catastrophic forgetting.

Contribution

GORP is a new training strategy that expands the optimization space by jointly updating full and low-rank parameters within a unified low-rank gradient subspace.

Findings

GORP outperforms existing methods on continual learning benchmarks.

It effectively mitigates catastrophic forgetting.

Maintains efficiency comparable to low-rank approaches.

Abstract

Continual fine-tuning of Large Language Models (LLMs) is hampered by the trade-off between efficiency and expressiveness. Low-Rank Adaptation (LoRA) offers efficiency but constrains the model's ability to learn new tasks and transfer knowledge due to its low-rank nature and reliance on explicit parameter constraints. We propose GORP (Gradient LOw Rank Projection) for Continual Learning, a novel training strategy that overcomes these limitations by synergistically combining full and low-rank parameters and jointly updating within a unified low-rank gradient subspace. GORP expands the optimization space while preserving efficiency and mitigating catastrophic forgetting. Extensive experiments on continual learning benchmarks demonstrate GORP's superior performance compared to existing state-of-the-art approaches. Code is available at https://github.com/Wcxwcxw/GORP.