Summarizing Stream Data for Memory-Constrained Online Continual Learning

TL;DR

This paper introduces a method called SSD that summarizes stream data into more informative samples for memory-efficient online continual learning, significantly improving replay effectiveness with minimal computational overhead.

Contribution

The paper proposes a novel data summarization technique that distills training characteristics into representative samples, enhancing replay in memory-constrained online continual learning.

Findings

SSD improves accuracy by over 3% on CIFAR-100 with limited memory.

Summarized samples better represent stream data than original images.

The method adds minimal computational overhead.

Abstract

Replay-based methods have proved their effectiveness on online continual learning by rehearsing past samples from an auxiliary memory. With many efforts made on improving training schemes based on the memory, however, the information carried by each sample in the memory remains under-investigated. Under circumstances with restricted storage space, the informativeness of the memory becomes critical for effective replay. Although some works design specific strategies to select representative samples, by only employing a small number of original images, the storage space is still not well utilized. To this end, we propose to Summarize the knowledge from the Stream Data (SSD) into more informative samples by distilling the training characteristics of real images. Through maintaining the consistency of training gradients and relationship to the past tasks, the summarized samples are more representative for the stream data compared to the original images. Extensive experiments are conducted on multiple online continual learning benchmarks to support that the proposed SSD method significantly enhances the replay effects. We demonstrate that with limited extra computational overhead, SSD provides more than 3% accuracy boost for sequential CIFAR-100 under extremely restricted memory buffer. Code in https://github.com/vimar-gu/SSD.