Memory-efficient Continual Learning with Neural Collapse Contrastive

TL;DR

This paper introduces FNC^2 and HSD losses for continual learning, balancing soft and hard relationships in representations, reducing memory needs, and rivaling rehearsal methods without memory use.

Contribution

It proposes a novel contrastive learning framework that balances soft and hard relationships and introduces a distillation loss to mitigate forgetting in continual learning.

Findings

Outperforms state-of-the-art methods in continual learning.

Achieves competitive results without memory, addressing privacy concerns.

Effectively balances intra-class variability and class separation.

Abstract

Contrastive learning has significantly improved representation quality, enhancing knowledge transfer across tasks in continual learning (CL). However, catastrophic forgetting remains a key challenge, as contrastive based methods primarily focus on "soft relationships" or "softness" between samples, which shift with changing data distributions and lead to representation overlap across tasks. Recently, the newly identified Neural Collapse phenomenon has shown promise in CL by focusing on "hard relationships" or "hardness" between samples and fixed prototypes. However, this approach overlooks "softness", crucial for capturing intra-class variability, and this rigid focus can also pull old class representations toward current ones, increasing forgetting. Building on these insights, we propose Focal Neural Collapse Contrastive (FNC^2), a novel representation learning loss that effectively balances both soft and hard relationships. Additionally, we introduce the Hardness-Softness Distillation (HSD) loss to progressively preserve the knowledge gained from these relationships across tasks. Our method outperforms state-of-the-art approaches, particularly in minimizing memory reliance. Remarkably, even without the use of memory, our approach rivals rehearsal-based methods, offering a compelling solution for data privacy concerns.