Leveraging Complementary Embeddings for Replay Selection in Continual Learning with Small Buffers

TL;DR

This paper introduces MERS, a graph-based replay selection method that combines supervised and self-supervised embeddings, significantly improving continual learning performance under memory constraints.

Contribution

MERS is a novel replay selection approach that leverages multiple embeddings, enhancing performance without extra parameters or increased memory use.

Findings

MERS outperforms state-of-the-art selection strategies on CIFAR-100 and TinyImageNet.

It achieves strong gains especially in low-memory scenarios.

MERS improves continual learning without additional model parameters.

Abstract

Catastrophic forgetting remains a key challenge in Continual Learning (CL). In replay-based CL with severe memory constraints, performance critically depends on the sample selection strategy for the replay buffer. Most existing approaches construct memory buffers using embeddings learned under supervised objectives. However, class-agnostic, self-supervised representations often encode rich, class-relevant semantics that are overlooked. We propose a new method, Multiple Embedding Replay Selection, MERS, which replaces the buffer selection module with a graph-based approach that integrates both supervised and self-supervised embeddings. Empirical results show consistent improvements over SOTA selection strategies across a range of continual learning algorithms, with particularly strong gains in low-memory regimes. On CIFAR-100 and TinyImageNet, MERS outperforms single-embedding baselines without adding model parameters or increasing replay volume, making it a practical, drop-in enhancement for replay-based continual learning.