Enhancing Pretrained Model-based Continual Representation Learning via Guided Random Projection

TL;DR

This paper introduces SCL-MGSM, a data-guided method for constructing a more expressive and stable random projection layer in continual learning, significantly improving performance over existing random projection approaches.

Contribution

It proposes a novel data-guided projection layer construction mechanism, enhancing expressivity and stability in continual learning with pre-trained models.

Findings

Outperforms state-of-the-art methods on CIL benchmarks

Achieves more stable and expressive feature representations

Reduces negative effects of domain gaps in continual learning

Abstract

Recent paradigms in Random Projection Layer (RPL)-based continual representation learning have demonstrated superior performance when building upon a pre-trained model (PTM). These methods insert a randomly initialized RPL after a PTM to enhance feature representation in the initial stage. Subsequently, a linear classification head is used for analytic updates in the continual learning stage. However, under severe domain gaps between pre-trained representations and target domains, a randomly initialized RPL exhibits limited expressivity under large domain shifts. While largely scaling up the RPL dimension can improve expressivity, it also induces an ill-conditioned feature matrix, thereby destabilizing the recursive analytic updates of the linear head. To this end, we propose the Stochastic Continual Learner with MemoryGuard Supervisory Mechanism (SCL-MGSM). Unlike random initialization, MGSM constructs the projection layer via a principled, data-guided mechanism that progressively selects target-aligned random bases to adapt the PTM representation to downstream tasks. This facilitates the construction of a compact yet expressive RPL while improving the numerical stability of analytic updates. Extensive experiments on multiple exemplar-free Class Incremental Learning (CIL) benchmarks demonstrate that SCL-MGSM achieves superior performance compared to state-of-the-art methods.