SPeCiaL: Self-Supervised Pretraining for Continual Learning

TL;DR

SPeCiaL introduces a self-supervised pretraining method for continual learning that enhances quick knowledge retention and minimizes forgetting, outperforming supervised pretraining in continual few-shot tasks.

Contribution

It proposes a novel meta-learning objective for unsupervised pretraining tailored to continual learning, differentiating through sequential processes to improve representation retention.

Findings

Matches or outperforms supervised pretraining in continual few-shot learning.

Produces representations that favor quick knowledge retention with minimal forgetting.

Effective in sequential learning scenarios.

Abstract

This paper presents SPeCiaL: a method for unsupervised pretraining of representations tailored for continual learning. Our approach devises a meta-learning objective that differentiates through a sequential learning process. Specifically, we train a linear model over the representations to match different augmented views of the same image together, each view presented sequentially. The linear model is then evaluated on both its ability to classify images it just saw, and also on images from previous iterations. This gives rise to representations that favor quick knowledge retention with minimal forgetting. We evaluate SPeCiaL in the Continual Few-Shot Learning setting, and show that it can match or outperform other supervised pretraining approaches.