Understanding Imbalanced Forgetting in Rehearsal-Based Class-Incremental Learning

TL;DR

This paper investigates the phenomenon of imbalanced forgetting in rehearsal-based class-incremental learning, identifying gradient-level factors that predict class-specific forgetting and suggesting avenues for mitigation.

Contribution

It introduces three last-layer coefficients derived from gradient analysis that predict class forgetting, offering a mechanistic understanding of imbalanced forgetting in CIL.

Findings

Identifies three gradient-based coefficients predicting class forgetting.

Shows one coefficient related to self-induced interference is the strongest predictor.

Provides insights into reducing class-wise disparities to mitigate forgetting.

Abstract

Neural networks suffer from catastrophic forgetting in class-incremental learning (CIL) settings. Rehearsal$\unicode{x2013}$replaying a subset of past samples$\unicode{x2013}$is a well-established mitigation strategy. However, recent results suggest that, despite balanced rehearsal allocation, some classes are forgotten substantially more than others. Despite its relevance, this imbalanced forgetting phenomenon remains underexplored. This work shows that imbalanced forgetting arises systematically and severely in rehearsal-based CIL and investigates it extensively. Specifically, we construct, from a principled analysis, three last-layer coefficients that capture different gradient-level sources of interference affecting each past class during an incremental step. We then demonstrate that, together, they reliably predict how past classes will rank in terms of forgetting at the end of that step. While predictive performance alone does not establish causality, these results support the interpretation of the coefficients as a plausible mechanistic account linking last-layer gradient-level interactions during training to class-level forgetting outcomes. Notably, one coefficient$\unicode{x2013}$capturing self-induced interference$\unicode{x2013}$emerges as the strongest predictor, with controlled experiments providing evidence consistent with this coefficient being influenced by the new-class interference coefficient. Overall, our findings provide valuable insights and suggest promising directions for mitigating imbalanced forgetting by reducing class-wise disparities in the identified sources of interference.