Learning to Imagine: Diversify Memory for Incremental Learning using Unlabeled Data

TL;DR

This paper introduces a method to mitigate catastrophic forgetting in incremental learning by generating diverse, semantically consistent samples from unlabeled data, enhancing memory diversity without extra inference cost.

Contribution

It proposes a learnable feature generator and semantic contrastive learning to diversify exemplars using unlabeled data, improving incremental learning performance.

Findings

Outperforms state-of-the-art on CIFAR-100 and ImageNet-Subset

Generates diverse, semantically consistent samples

No extra inference cost incurred

Abstract

Deep neural network (DNN) suffers from catastrophic forgetting when learning incrementally, which greatly limits its applications. Although maintaining a handful of samples (called `exemplars`) of each task could alleviate forgetting to some extent, existing methods are still limited by the small number of exemplars since these exemplars are too few to carry enough task-specific knowledge, and therefore the forgetting remains. To overcome this problem, we propose to `imagine` diverse counterparts of given exemplars referring to the abundant semantic-irrelevant information from unlabeled data. Specifically, we develop a learnable feature generator to diversify exemplars by adaptively generating diverse counterparts of exemplars based on semantic information from exemplars and semantically-irrelevant information from unlabeled data. We introduce semantic contrastive learning to enforce the generated samples to be semantic consistent with exemplars and perform semanticdecoupling contrastive learning to encourage diversity of generated samples. The diverse generated samples could effectively prevent DNN from forgetting when learning new tasks. Our method does not bring any extra inference cost and outperforms state-of-the-art methods on two benchmarks CIFAR-100 and ImageNet-Subset by a clear margin.