Expandable and Differentiable Dual Memories with Orthogonal Regularization for Exemplar-free Continual Learning

TL;DR

This paper introduces a novel exemplar-free continual learning method with dual differentiable memories and orthogonal regularization, enabling better task knowledge integration and avoiding interference, thus outperforming existing methods on multiple datasets.

Contribution

It proposes a fully differentiable dual-memory architecture with adaptive capacity expansion and orthogonal regularization for exemplar-free continual learning, improving knowledge transfer and task separation.

Findings

Outperforms 14 state-of-the-art methods on CIFAR-10, CIFAR-100, and Tiny-ImageNet.

Achieves higher final accuracies and better feature extraction close to the upper bound.

Effectively prevents interference between learned tasks through orthogonal regularization.

Abstract

Continual learning methods used to force neural networks to process sequential tasks in isolation, preventing them from leveraging useful inter-task relationships and causing them to repeatedly relearn similar features or overly differentiate them. To address this problem, we propose a fully differentiable, exemplar-free expandable method composed of two complementary memories: One learns common features that can be used across all tasks, and the other combines the shared features to learn discriminative characteristics unique to each sample. Both memories are differentiable so that the network can autonomously learn latent representations for each sample. For each task, the memory adjustment module adaptively prunes critical slots and minimally expands capacity to accommodate new concepts, and orthogonal regularization enforces geometric separation between preserved and newly learned memory components to prevent interference. Experiments on CIFAR-10, CIFAR-100, and Tiny-ImageNet show that the proposed method outperforms 14 state-of-the-art methods for class-incremental learning, achieving final accuracies of 55.13\%, 37.24\%, and 30.11\%, respectively. Additional analysis confirms that, through effective integration and utilization of knowledge, the proposed method can increase average performance across sequential tasks, and it produces feature extraction results closest to the upper bound, thus establishing a new milestone in continual learning.