Generative Kernel Continual learning

TL;DR

This paper introduces generative kernel continual learning, combining generative models with kernels to eliminate memory dependence, improve efficiency, and enhance classification performance in continual learning tasks.

Contribution

It proposes a novel approach that uses generative models to produce samples for kernel learning, reducing memory needs and task interference while boosting accuracy.

Findings

Achieves comparable accuracy to memory-intensive methods with less memory.

Demonstrates significant accuracy gains on SplitCIFAR100.

Provides a scalable, efficient continual learning framework.

Abstract

Kernel continual learning by \citet{derakhshani2021kernel} has recently emerged as a strong continual learner due to its non-parametric ability to tackle task interference and catastrophic forgetting. Unfortunately its success comes at the expense of an explicit memory to store samples from past tasks, which hampers scalability to continual learning settings with a large number of tasks. In this paper, we introduce generative kernel continual learning, which explores and exploits the synergies between generative models and kernels for continual learning. The generative model is able to produce representative samples for kernel learning, which removes the dependence on memory in kernel continual learning. Moreover, as we replay only on the generative model, we avoid task interference while being computationally more efficient compared to previous methods that need replay on the entire model. We further introduce a supervised contrastive regularization, which enables our model to generate even more discriminative samples for better kernel-based classification performance. We conduct extensive experiments on three widely-used continual learning benchmarks that demonstrate the abilities and benefits of our contributions. Most notably, on the challenging SplitCIFAR100 benchmark, with just a simple linear kernel we obtain the same accuracy as kernel continual learning with variational random features for one tenth of the memory, or a 10.1\% accuracy gain for the same memory budget.