Continual Learning Beyond Experience Rehearsal and Full Model Surrogates

TL;DR

SPARC is a scalable continual learning method that combines task-specific and semantic memories, achieving high efficiency and competitive performance without experience rehearsal or full model surrogates.

Contribution

It introduces SPARC, a novel continual learning approach that reduces memory and computational costs by eliminating experience rehearsal and full model surrogates.

Findings

Uses only 6% of parameters compared to full-model surrogates.

Achieves superior performance on Seq-TinyImageNet.

Matches rehearsal-based methods on various benchmarks.

Abstract

Continual learning (CL) has remained a significant challenge for deep neural networks as learning new tasks erases previously acquired knowledge, either partially or completely. Existing solutions often rely on experience rehearsal or full model surrogates to mitigate CF. While effective, these approaches introduce substantial memory and computational overhead, limiting their scalability and applicability in real-world scenarios. To address this, we propose SPARC, a scalable CL approach that eliminates the need for experience rehearsal and full-model surrogates. By effectively combining task-specific working memories and task-agnostic semantic memory for cross-task knowledge consolidation, SPARC results in a remarkable parameter efficiency, using only 6% of the parameters required by full-model surrogates. Despite its lightweight design, SPARC achieves superior performance on Seq-TinyImageNet and matches rehearsal-based methods on various CL benchmarks. Additionally, weight re-normalization in the classification layer mitigates task-specific biases, establishing SPARC as a practical and scalable solution for CL under stringent efficiency constraints.