Enhancing Consistency and Mitigating Bias: A Data Replay Approach for Incremental Learning

TL;DR

This paper introduces CwD, a data-free replay method that reduces inconsistency between inverted and real data and balances class weights, significantly improving incremental learning performance without extra memory.

Contribution

It proposes a novel loss function based on KL divergence under Gaussian assumptions and a regularization term to balance class weights, enhancing data replay in continual learning.

Findings

CwD outperforms previous methods on CIFAR-100, Tiny-ImageNet, and ImageNet100.

The proposed loss reduces data inconsistency effectively.

Class weight regularization improves old class distinguishability.

Abstract

Deep learning systems are prone to catastrophic forgetting when learning from a sequence of tasks, as old data from previous tasks is unavailable when learning a new task. To address this, some methods propose replaying data from previous tasks during new task learning, typically using extra memory to store replay data. However, it is not expected in practice due to memory constraints and data privacy issues. Instead, data-free replay methods invert samples from the classification model. While effective, these methods face inconsistencies between inverted and real training data, overlooked in recent works. To that effect, we propose to measure the data consistency quantitatively by some simplification and assumptions. Using this measurement, we gain insight to develop a novel loss function that reduces inconsistency. Specifically, the loss minimizes the KL divergence between distributions of inverted and real data under a tied multivariate Gaussian assumption, which is simple to implement in continual learning. Additionally, we observe that old class weight norms decrease continually as learning progresses. We analyze the reasons and propose a regularization term to balance class weights, making old class samples more distinguishable. To conclude, we introduce Consistency-enhanced data replay with a Debiased classifier for class incremental learning (CwD). Extensive experiments on CIFAR-100, Tiny-ImageNet, and ImageNet100 show consistently improved performance of CwD compared to previous approaches.