Continual Learning with Deep Streaming Regularized Discriminant Analysis

TL;DR

This paper introduces a streaming regularized discriminant analysis method integrated with CNNs to enable continual learning from streaming data, effectively mitigating catastrophic forgetting and outperforming existing methods on ImageNet.

Contribution

We propose a novel streaming regularized discriminant analysis algorithm combined with CNNs for effective continual learning from streaming data.

Findings

Outperforms batch learning on ImageNet

Outperforms existing streaming learning algorithms

Mitigates catastrophic forgetting in streaming scenarios

Abstract

Continual learning is increasingly sought after in real world machine learning applications, as it enables learning in a more human-like manner. Conventional machine learning approaches fail to achieve this, as incrementally updating the model with non-identically distributed data leads to catastrophic forgetting, where existing representations are overwritten. Although traditional continual learning methods have mostly focused on batch learning, which involves learning from large collections of labeled data sequentially, this approach is not well-suited for real-world applications where we would like new data to be integrated directly. This necessitates a paradigm shift towards streaming learning. In this paper, we propose a streaming version of regularized discriminant analysis as a solution to this challenge. We combine our algorithm with a convolutional neural network and demonstrate that it outperforms both batch learning and existing streaming learning algorithms on the ImageNet ILSVRC-2012 dataset.