ELLA: Efficient Lifelong Learning for Adapters in Large Language Models

TL;DR

ELLA is a scalable lifelong learning framework for large language models that mitigates catastrophic forgetting without replay or expansion, using a novel subspace de-correlation regularizer to enable transfer and improve performance.

Contribution

ELLA introduces a new regularization-based method for lifelong learning in LLMs that selectively de-correlates task-specific updates, outperforming existing methods in efficiency and accuracy.

Findings

Achieves state-of-the-art continual learning performance on benchmarks.

Reduces memory footprint by up to 35 times compared to previous methods.

Enhances zero-shot generalization to unseen tasks.

Abstract

Large Language Models (LLMs) suffer severe catastrophic forgetting when adapted sequentially to new tasks in a continual learning (CL) setting. Existing approaches are fundamentally limited: replay-based methods are impractical and privacy-violating, while strict orthogonality-based methods collapse under scale: each new task is projected onto an orthogonal complement, progressively reducing the residual degrees of freedom and eliminating forward transfer by forbidding overlap in shared representations. In this work, we introduce ELLA, a training framework built on the principle of selective subspace de-correlation. Rather than forbidding all overlap, ELLA explicitly characterizes the structure of past updates and penalizes alignments along their high-energy, task-specific directions, while preserving freedom in the low-energy residual subspaces to enable transfer. Formally, this is realized via a lightweight regularizer on a single aggregated update matrix. We prove this mechanism corresponds to an anisotropic shrinkage operator that bounds interference, yielding a penalty that is both memory- and compute-constant regardless of task sequence length. ELLA requires no data replay, no architectural expansion, and negligible storage. Empirically, it achieves state-of-the-art CL performance on three popular benchmarks, with relative accuracy gains of up to $9.6\%$ and a $35\times$ smaller memory footprint. Further, ELLA scales robustly across architectures and actively enhances the model's zero-shot generalization performance on unseen tasks, establishing a principled and scalable solution for constructive lifelong LLM adaptation.