Contrastive Consolidation of Top-Down Modulations Achieves Sparsely Supervised Continual Learning

TL;DR

This paper introduces a biologically inspired contrastive learning method, TMCL, that enables continual learning with minimal supervision by integrating top-down modulations to prevent catastrophic forgetting.

Contribution

The paper proposes TMCL, a novel contrastive learning framework inspired by neocortical mechanisms, for sparsely supervised continual learning without degrading previous task performance.

Findings

Improves class-incremental learning performance with as little as 1% labels.

Outperforms state-of-the-art unsupervised and supervised methods in transfer learning.

Stabilizes representations using modulation invariance and past modulations.

Abstract

Biological brains learn continually from a stream of unlabeled data, while integrating specialized information from sparsely labeled examples without compromising their ability to generalize. Meanwhile, machine learning methods are susceptible to catastrophic forgetting in this natural learning setting, as supervised specialist fine-tuning degrades performance on the original task. We introduce task-modulated contrastive learning (TMCL), which takes inspiration from the biophysical machinery in the neocortex, using predictive coding principles to integrate top-down information continually and without supervision. We follow the idea that these principles build a view-invariant representation space, and that this can be implemented using a contrastive loss. Then, whenever labeled samples of a new class occur, new affine modulations are learned that improve separation of the new class from all others, without affecting feedforward weights. By co-opting the view-invariance learning mechanism, we then train feedforward weights to match the unmodulated representation of a data sample to its modulated counterparts. This introduces modulation invariance into the representation space, and, by also using past modulations, stabilizes it. Our experiments show improvements in both class-incremental and transfer learning over state-of-the-art unsupervised approaches, as well as over comparable supervised approaches, using as few as 1% of available labels. Taken together, our work suggests that top-down modulations play a crucial role in balancing stability and plasticity.