Task Confusion and Catastrophic Forgetting in Class-Incremental Learning: A Mathematical Framework for Discriminative and Generative Modelings

TL;DR

This paper introduces a mathematical framework for class-incremental learning, proving that generative models can overcome task confusion and achieve optimal performance, unlike discriminative models.

Contribution

It establishes the Infeasibility Theorem and Feasibility Theorem, providing a theoretical foundation for the advantages of generative modeling in class-IL.

Findings

Discriminative models are limited by task confusion in class-IL.

Generative models can theoretically achieve optimal class-IL.

Analysis of popular strategies highlights the importance of generative approaches.

Abstract

In class-incremental learning (class-IL), models must classify all previously seen classes at test time without task-IDs, leading to task confusion. Despite being a key challenge, task confusion lacks a theoretical understanding. We present a novel mathematical framework for class-IL and prove the Infeasibility Theorem, showing optimal class-IL is impossible with discriminative modeling due to task confusion. However, we establish the Feasibility Theorem, demonstrating that generative modeling can achieve optimal class-IL by overcoming task confusion. We then assess popular class-IL strategies, including regularization, bias-correction, replay, and generative classifier, using our framework. Our analysis suggests that adopting generative modeling, either for generative replay or direct classification (generative classifier), is essential for optimal class-IL.