Compact Memory for Continual Logistic Regression

TL;DR

This paper introduces a novel method for building compact memory in continual logistic regression, significantly improving accuracy and reducing memory requirements compared to existing experience replay techniques.

Contribution

We propose a Hessian-matching based probabilistic PCA approach to optimize memory for logistic regression in continual learning, achieving near-batch performance with minimal memory.

Findings

Achieves 60% accuracy on Split-ImageNet with 0.3% memory size

Boosts accuracy to 74% with 2% memory size, close to batch accuracy of 77.6%

Outperforms traditional experience replay in accuracy and memory efficiency

Abstract

Despite recent progress, continual learning still does not match the performance of batch training. To avoid catastrophic forgetting, we need to build compact memory of essential past knowledge, but no clear solution has yet emerged, even for shallow neural networks with just one or two layers. In this paper, we present a new method to build compact memory for logistic regression. Our method is based on a result by Khan and Swaroop [2021] who show the existence of optimal memory for such models. We formulate the search for the optimal memory as Hessian-matching and propose a probabilistic PCA method to estimate them. Our approach can drastically improve accuracy compared to Experience Replay. For instance, on Split-ImageNet, we get 60% accuracy compared to 30% obtained by replay with memory-size equivalent to 0.3% of the data size. Increasing the memory size to 2% further boosts the accuracy to 74%, closing the gap to the batch accuracy of 77.6% on this task. Our work opens a new direction for building compact memory that can also be useful in the future for continual deep learning.