On the Effectiveness of Equivariant Regularization for Robust Online Continual Learning

TL;DR

This paper introduces CLER, a novel online continual learning method that uses equivariant regularization with self-supervision to improve knowledge retention and transfer, addressing the limitations of contrastive self-supervised learning in streaming data scenarios.

Contribution

It presents the first integration of equivariant knowledge with continual learning, enhancing online CL performance and compatibility with existing methods.

Findings

Equivariant pretext tasks influence network information flow.

CLER improves knowledge retention in online continual learning.

The method is easily integrated with existing OCL approaches.

Abstract

Humans can learn incrementally, whereas neural networks forget previously acquired information catastrophically. Continual Learning (CL) approaches seek to bridge this gap by facilitating the transfer of knowledge to both previous tasks (backward transfer) and future ones (forward transfer) during training. Recent research has shown that self-supervision can produce versatile models that can generalize well to diverse downstream tasks. However, contrastive self-supervised learning (CSSL), a popular self-supervision technique, has limited effectiveness in online CL (OCL). OCL only permits one iteration of the input dataset, and CSSL's low sample efficiency hinders its use on the input data-stream. In this work, we propose Continual Learning via Equivariant Regularization (CLER), an OCL approach that leverages equivariant tasks for self-supervision, avoiding CSSL's limitations. Our method represents the first attempt at combining equivariant knowledge with CL and can be easily integrated with existing OCL methods. Extensive ablations shed light on how equivariant pretext tasks affect the network's information flow and its impact on CL dynamics.