TL;DR
This paper investigates the causes of catastrophic forgetting in class-incremental learning and proposes simple, effective solutions that significantly improve performance on CIFAR-100 and ImageNet.
Contribution
It demonstrates that combining simple components with a balanced loss and regularization effectively mitigates forgetting, surpassing complex existing methods.
Findings
Simple components and balanced loss reduce forgetting.
Representation quality correlates with secondary class information.
Performance on CIFAR-100 and ImageNet improves significantly.
Abstract
Contemporary neural networks are limited in their ability to learn from evolving streams of training data. When trained sequentially on new or evolving tasks, their accuracy drops sharply, making them unsuitable for many real-world applications. In this work, we shed light on the causes of this well-known yet unsolved phenomenon - often referred to as catastrophic forgetting - in a class-incremental setup. We show that a combination of simple components and a loss that balances intra-task and inter-task learning can already resolve forgetting to the same extent as more complex measures proposed in literature. Moreover, we identify poor quality of the learned representation as another reason for catastrophic forgetting in class-IL. We show that performance is correlated with secondary class information (dark knowledge) learned by the model and it can be improved by an appropriate…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
