A comprehensive, application-oriented study of catastrophic forgetting in DNNs

TL;DR

This paper conducts a large-scale empirical study on catastrophic forgetting in deep neural networks during sequential learning, revealing that no existing model completely avoids CF across diverse datasets and tasks.

Contribution

It introduces a new experimental protocol for realistic application scenarios and evaluates CF across the largest set of visual datasets to date.

Findings

No model completely avoids CF across all datasets and tasks

EWC and IMM models have potential workarounds for CF

Empirical evidence highlights the challenge of mitigating CF in practical settings

Abstract

We present a large-scale empirical study of catastrophic forgetting (CF) in modern Deep Neural Network (DNN) models that perform sequential (or: incremental) learning. A new experimental protocol is proposed that enforces typical constraints encountered in application scenarios. As the investigation is empirical, we evaluate CF behavior on the hitherto largest number of visual classification datasets, from each of which we construct a representative number of Sequential Learning Tasks (SLTs) in close alignment to previous works on CF. Our results clearly indicate that there is no model that avoids CF for all investigated datasets and SLTs under application conditions. We conclude with a discussion of potential solutions and workarounds to CF, notably for the EWC and IMM models.