Catastrophic Forgetting in Deep Learning: A Comprehensive Taxonomy

TL;DR

This paper provides a comprehensive review and taxonomy of methods addressing catastrophic forgetting in deep learning, highlighting current solutions, research gaps, and the ongoing challenge of enabling models to learn incrementally without losing prior knowledge.

Contribution

It offers a detailed taxonomy of recent approaches to mitigate catastrophic forgetting and identifies key research gaps in the field.

Findings

Survey of recent solutions to CF in deep learning

Proposed taxonomy organizes existing methods

Highlights unresolved challenges and research gaps

Abstract

Deep Learning models have achieved remarkable performance in tasks such as image classification or generation, often surpassing human accuracy. However, they can struggle to learn new tasks and update their knowledge without access to previous data, leading to a significant loss of accuracy known as Catastrophic Forgetting (CF). This phenomenon was first observed by McCloskey and Cohen in 1989 and remains an active research topic. Incremental learning without forgetting is widely recognized as a crucial aspect in building better AI systems, as it allows models to adapt to new tasks without losing the ability to perform previously learned ones. This article surveys recent studies that tackle CF in modern Deep Learning models that use gradient descent as their learning algorithm. Although several solutions have been proposed, a definitive solution or consensus on assessing CF is yet to be established. The article provides a comprehensive review of recent solutions, proposes a taxonomy to organize them, and identifies research gaps in this area.