Hybrid Memory Replay: Blending Real and Distilled Data for Class Incremental Learning

TL;DR

This paper introduces a hybrid memory approach combining real and synthetic data for class incremental learning, effectively reducing catastrophic forgetting with limited buffer sizes.

Contribution

It proposes a novel hybrid memory system with a modified data distillation process and real exemplar selection, improving replay-based incremental learning performance.

Findings

Significantly outperforms existing replay-based methods across benchmarks.

Effectively mitigates catastrophic forgetting with limited buffer sizes.

Seamlessly integrates into most existing replay-based CIL models.

Abstract

Incremental learning (IL) aims to acquire new knowledge from current tasks while retaining knowledge learned from previous tasks. Replay-based IL methods store a set of exemplars from previous tasks in a buffer and replay them when learning new tasks. However, there is usually a size-limited buffer that cannot store adequate real exemplars to retain the knowledge of previous tasks. In contrast, data distillation (DD) can reduce the exemplar buffer's size, by condensing a large real dataset into a much smaller set of more information-compact synthetic exemplars. Nevertheless, DD's performance gain on IL quickly vanishes as the number of synthetic exemplars grows. To overcome the weaknesses of real-data and synthetic-data buffers, we instead optimize a hybrid memory including both types of data. Specifically, we propose an innovative modification to DD that distills synthetic data from a sliding window of checkpoints in history (rather than checkpoints on multiple training trajectories). Conditioned on the synthetic data, we then optimize the selection of real exemplars to provide complementary improvement to the DD objective. The optimized hybrid memory combines the strengths of synthetic and real exemplars, effectively mitigating catastrophic forgetting in Class IL (CIL) when the buffer size for exemplars is limited. Notably, our method can be seamlessly integrated into most existing replay-based CIL models. Extensive experiments across multiple benchmarks demonstrate that our method significantly outperforms existing replay-based baselines.