CroMo-Mixup: Augmenting Cross-Model Representations for Continual Self-Supervised Learning

TL;DR

CroMo-Mixup introduces a novel framework that enhances continual self-supervised learning by addressing task confusion through cross-model and cross-task data Mixup techniques, improving task identification and accuracy.

Contribution

The paper proposes CroMo-Mixup, a new method combining cross-task data Mixup and cross-model feature Mixup to mitigate task confusion in CSSL.

Findings

Improves task ID prediction accuracy.

Enhances average linear accuracy across tasks.

Compatible with multiple SSL objectives.

Abstract

Continual self-supervised learning (CSSL) learns a series of tasks sequentially on the unlabeled data. Two main challenges of continual learning are catastrophic forgetting and task confusion. While CSSL problem has been studied to address the catastrophic forgetting challenge, little work has been done to address the task confusion aspect. In this work, we show through extensive experiments that self-supervised learning (SSL) can make CSSL more susceptible to the task confusion problem, particularly in less diverse settings of class incremental learning because different classes belonging to different tasks are not trained concurrently. Motivated by this challenge, we present a novel cross-model feature Mixup (CroMo-Mixup) framework that addresses this issue through two key components: 1) Cross-Task data Mixup, which mixes samples across tasks to enhance negative sample diversity; and 2) Cross-Model feature Mixup, which learns similarities between embeddings obtained from current and old models of the mixed sample and the original images, facilitating cross-task class contrast learning and old knowledge retrieval. We evaluate the effectiveness of CroMo-Mixup to improve both Task-ID prediction and average linear accuracy across all tasks on three datasets, CIFAR10, CIFAR100, and tinyImageNet under different class-incremental learning settings. We validate the compatibility of CroMo-Mixup on four state-of-the-art SSL objectives. Code is available at \url{https://github.com/ErumMushtaq/CroMo-Mixup}.