Adaptive Budget Allocation for Orthogonal-Subspace Adapter Tuning in LLMs Continual Learning

TL;DR

This paper introduces OA-Adapter, a novel method for continual learning in large language models that dynamically allocates parameter budgets and applies orthogonal constraints to mitigate forgetting and improve efficiency.

Contribution

OA-Adapter unifies dynamic budget adaptation with orthogonal subspace learning in an end-to-end training process for LLMs in continual learning.

Findings

Outperforms state-of-the-art methods in accuracy.

Uses 58.5% fewer parameters on standard benchmarks.

Maintains advantages on larger, multi-task benchmarks.

Abstract

Large language models (LLMs) often suffer from catastrophic forgetting in continual learning (CL) scenarios, where performance on previously learned tasks degrades severely while training on sequentially arriving tasks. Although pioneering CL approaches using orthogonal subspaces can mitigate task interference, they typically employ fixed budget allocation, neglecting the varying complexity across tasks and layers. Besides, recent budget-adaptive tuning methods for LLMs often adopt multi-stage paradigms that decouple optimization and budget allocation. Such decoupling results in potential misalignment, which hinders those approaches' practical application in CL scenarios. To address these limitations, we propose OA-Adapter, a novel parameter-efficient approach for continual learning in LLMs that unifies dynamic budget adaptation with orthogonal subspace learning in an end-to-end training stage. Specifically, OA-Adapter introduces a dynamic bottleneck dimension adaptation mechanism that simultaneously allocates an efficient parameter budget and optimizes task objectives without misalignment.To effectively preserve previously acquired knowledge while coordinating with the dynamic budget allocation, orthogonal constraints are applied specifically between the parameter subspace of the current task and the dynamically allocated parameter subspaces of historical tasks. Experimental results on continual learning benchmarks demonstrate that OA-Adapter outperforms state-of-the-art methods in both accuracy and parameter efficiency. OA-Adapter achieves higher average accuracy while using 58.5% fewer parameters on the standard CL benchmark, and maintains its advantages on two larger benchmarks comprising 15 tasks.