Gated Integration of Low-Rank Adaptation for Continual Learning of Large Language Models

TL;DR

GainLoRA is a novel continual learning method for large language models that uses gated integration of low-rank adaptation branches to reduce forgetting and enhance performance across multiple tasks.

Contribution

The paper introduces GainLoRA, a new approach that expands LoRA branches per task and employs gating modules to effectively mitigate forgetting in continual learning.

Findings

GainLoRA outperforms existing methods on CL benchmarks.

Gating modules effectively minimize influence from new to old tasks.

The method improves overall continual learning performance.

Abstract

Continual learning (CL), which requires the model to learn multiple tasks sequentially, is crucial for large language models (LLMs). Recently, low-rank adaptation~(LoRA), one of the most representative parameter-efficient fine-tuning (PEFT) methods, has gained increasing attention in CL of LLMs. However, most existing CL methods based on LoRA typically expand a new LoRA branch to learn each new task and force the new and old LoRA branches to influence old tasks equally, potentially leading to forgetting. In this work, we propose a new method, called gated integration of low-rank adaptation (GainLoRA), for CL of LLMs. GainLoRA expands a new LoRA branch for each new task and introduces gating modules to integrate the new and old LoRA branches. Furthermore, GainLoRA leverages the new gating module to minimize the influence from the new LoRA branch to old tasks, effectively mitigating forgetting and improving the model's overall performance. Experimental results on CL benchmarks demonstrate that GainLoRA outperforms existing state-of-the-art methods.