PLATE: Plasticity-Tunable Efficient Adapters for Geometry-Aware Continual Learning

TL;DR

PLATE is a continual learning approach that leverages geometric redundancy in pretrained models to adapt efficiently without access to old-task data, balancing plasticity and retention.

Contribution

It introduces a novel method exploiting pretrained model redundancy to enable data-free continual learning with explicit plasticity-retention control.

Findings

Requires no old-task data for continual learning.

Reduces functional drift and improves retention.

Provides explicit control over plasticity-retention trade-off.

Abstract

We develop a continual learning method for pretrained models that \emph{requires no access to old-task data}, addressing a practical barrier in foundation model adaptation where pretraining distributions are often unavailable. Our key observation is that pretrained networks exhibit substantial \emph{geometric redundancy}, and that this redundancy can be exploited in two complementary ways. First, redundant neurons provide a proxy for dominant pretraining-era feature directions, enabling the construction of approximately protected update subspaces directly from pretrained weights. Second, redundancy offers a natural bias for \emph{where} to place plasticity: by restricting updates to a subset of redundant neurons and constraining the remaining degrees of freedom, we obtain update families with reduced functional drift on the old-data distribution and improved worst-case retention guarantees. These insights lead to \textsc{PLATE} (\textbf{Pla}sticity-\textbf{T}unable \textbf{E}fficient Adapters), a continual learning method requiring no past-task data that provides explicit control over the plasticity-retention trade-off. PLATE parameterizes each layer with a structured low-rank update $\Delta W = B A Q^\top$, where $B$ and $Q$ are computed once from pretrained weights and kept frozen, and only $A$ is trained on the new task. The code is available at https://github.com/SalesforceAIResearch/PLATE.