Group and Exclusive Sparse Regularization-based Continual Learning of CNNs

TL;DR

This paper introduces GESCL, a regularization-based continual learning method for CNNs that prevents forgetting and efficiently sparsifies the network, outperforming existing approaches on vision benchmarks.

Contribution

The paper proposes GESCL, a novel regularization technique that stabilizes important filters and promotes sparsity, reducing parameters and computation in continual learning.

Findings

GESCL outperforms state-of-the-art methods in accuracy

GESCL effectively prevents catastrophic forgetting

GESCL reduces network parameters and computational cost

Abstract

We present a regularization-based approach for continual learning (CL) of fixed capacity convolutional neural networks (CNN) that does not suffer from the problem of catastrophic forgetting when learning multiple tasks sequentially. This method referred to as Group and Exclusive Sparsity based Continual Learning (GESCL) avoids forgetting of previous tasks by ensuring the stability of the CNN via a stability regularization term, which prevents filters detected as important for past tasks to deviate too much when learning a new task. On top of that, GESCL makes the network plastic via a plasticity regularization term that leverage the over-parameterization of CNNs to efficiently sparsify the network and tunes unimportant filters making them relevant for future tasks. Doing so, GESCL deals with significantly less parameters and computation compared to CL approaches that either dynamically expand the network or memorize past tasks' data. Experiments on popular CL vision benchmarks show that GESCL leads to significant improvements over state-of-the-art method in terms of overall CL performance, as measured by classification accuracy as well as in terms of avoiding catastrophic forgetting.