Incremental Learning via Rate Reduction

TL;DR

This paper introduces a 'white box' incremental learning method based on rate reduction that explicitly constructs networks to prevent catastrophic forgetting, outperforming existing approaches on MNIST and CIFAR-10.

Contribution

It proposes a novel 'white box' architecture derived from rate reduction principles that enables provable incremental learning without backpropagation.

Findings

Significantly less performance decay compared to state-of-the-art methods

Outperforms existing methods on MNIST and CIFAR-10 datasets

Provides a provable construction for incremental learning with pre-trained networks

Abstract

Current deep learning architectures suffer from catastrophic forgetting, a failure to retain knowledge of previously learned classes when incrementally trained on new classes. The fundamental roadblock faced by deep learning methods is that deep learning models are optimized as "black boxes," making it difficult to properly adjust the model parameters to preserve knowledge about previously seen data. To overcome the problem of catastrophic forgetting, we propose utilizing an alternative "white box" architecture derived from the principle of rate reduction, where each layer of the network is explicitly computed without back propagation. Under this paradigm, we demonstrate that, given a pre-trained network and new data classes, our approach can provably construct a new network that emulates joint training with all past and new classes. Finally, our experiments show that our proposed learning algorithm observes significantly less decay in classification performance, outperforming state of the art methods on MNIST and CIFAR-10 by a large margin and justifying the use of "white box" algorithms for incremental learning even for sufficiently complex image data.