Continual Learning Using a Kernel-Based Method Over Foundation Models

TL;DR

This paper introduces KLDA, a kernel-based continual learning method that leverages foundation model features with RBF kernels to effectively prevent catastrophic forgetting and inter-task class separation issues, achieving near joint training accuracy.

Contribution

The paper proposes KLDA, a novel kernel-based method for class-incremental learning that enhances foundation model features with RBF kernels and RFF, outperforming existing methods without replay data.

Findings

KLDA outperforms baseline methods on text and image datasets.

KLDA achieves accuracy comparable to joint training without replay data.

The method effectively mitigates catastrophic forgetting and class separation issues.

Abstract

Continual learning (CL) learns a sequence of tasks incrementally. This paper studies the challenging CL setting of class-incremental learning (CIL). CIL has two key challenges: catastrophic forgetting (CF) and inter-task class separation (ICS). Despite numerous proposed methods, these issues remain persistent obstacles. This paper proposes a novel CIL method, called Kernel Linear Discriminant Analysis (KLDA), that can effectively avoid CF and ICS problems. It leverages only the powerful features learned in a foundation model (FM). However, directly using these features proves suboptimal. To address this, KLDA incorporates the Radial Basis Function (RBF) kernel and its Random Fourier Features (RFF) to enhance the feature representations from the FM, leading to improved performance. When a new task arrives, KLDA computes only the mean for each class in the task and updates a shared covariance matrix for all learned classes based on the kernelized features. Classification is performed using Linear Discriminant Analysis. Our empirical evaluation using text and image classification datasets demonstrates that KLDA significantly outperforms baselines. Remarkably, without relying on replay data, KLDA achieves accuracy comparable to joint training of all classes, which is considered the upper bound for CIL performance. The KLDA code is available at https://github.com/salehmomeni/klda.