AIR: Analytic Imbalance Rectifier for Continual Learning

TL;DR

The paper introduces AIR, an online continual learning method with an analytic solution that effectively addresses data imbalance and class-incremental learning challenges, outperforming existing approaches.

Contribution

AIR provides a novel analytic re-weighting module and a least squares-based classifier for balanced continual learning without exemplars.

Findings

AIR outperforms existing methods in long-tailed scenarios.

AIR effectively handles generalized class-incremental learning.

The method is exemplar-free and computationally efficient.

Abstract

Continual learning enables AI models to learn new data sequentially without retraining in real-world scenarios. Most existing methods assume the training data are balanced, aiming to reduce the catastrophic forgetting problem that models tend to forget previously generated data. However, data imbalance and the mixture of new and old data in real-world scenarios lead the model to ignore categories with fewer training samples. To solve this problem, we propose an analytic imbalance rectifier algorithm (AIR), a novel online exemplar-free continual learning method with an analytic (i.e., closed-form) solution for data-imbalanced class-incremental learning (CIL) and generalized CIL scenarios in real-world continual learning. AIR introduces an analytic re-weighting module (ARM) that calculates a re-weighting factor for each class for the loss function to balance the contribution of each category to the overall loss and solve the problem of imbalanced training data. AIR uses the least squares technique to give a non-discriminatory optimal classifier and its iterative update method in continual learning. Experimental results on multiple datasets show that AIR significantly outperforms existing methods in long-tailed and generalized CIL scenarios. The source code is available at https://github.com/fang-d/AIR.