PANDA -- Patch And Distribution-Aware Augmentation for Long-Tailed Exemplar-Free Continual Learning

TL;DR

PANDA is a novel augmentation framework designed for exemplar-free continual learning that addresses data imbalance issues by enhancing low-frequency classes and balancing task distributions, thereby improving model accuracy and reducing forgetting.

Contribution

PANDA introduces a patch-based augmentation and adaptive balancing strategy that effectively mitigates intra-task and inter-task data imbalances in PTM-based EFCL.

Findings

PANDA improves accuracy across multiple EFCL benchmarks.

PANDA reduces catastrophic forgetting in continual learning.

PANDA seamlessly integrates with existing PTM-based methods.

Abstract

Exemplar-Free Continual Learning (EFCL) restricts the storage of previous task data and is highly susceptible to catastrophic forgetting. While pre-trained models (PTMs) are increasingly leveraged for EFCL, existing methods often overlook the inherent imbalance of real-world data distributions. We discovered that real-world data streams commonly exhibit dual-level imbalances, dataset-level distributions combined with extreme or reversed skews within individual tasks, creating both intra-task and inter-task disparities that hinder effective learning and generalization. To address these challenges, we propose PANDA, a Patch-and-Distribution-Aware Augmentation framework that integrates seamlessly with existing PTM-based EFCL methods. PANDA amplifies low-frequency classes by using a CLIP encoder to identify representative regions and transplanting those into frequent-class samples within each task. Furthermore, PANDA incorporates an adaptive balancing strategy that leverages prior task distributions to smooth inter-task imbalances, reducing the overall gap between average samples across tasks and enabling fairer learning with frozen PTMs. Extensive experiments and ablation studies demonstrate PANDA's capability to work with existing PTM-based CL methods, improving accuracy and reducing catastrophic forgetting.