Catastrophic Forgetting in the Context of Model Updates

TL;DR

This paper compares methods to mitigate catastrophic forgetting in model updates, highlighting that data rehearsal combined with techniques like EWC offers superior accuracy and efficiency when maintaining access to past data.

Contribution

It demonstrates that simple data rehearsal, especially when combined with EWC, effectively reduces forgetting and improves model update efficiency.

Findings

Data rehearsal outperforms other methods in preserving past knowledge.

Combining data rehearsal with EWC enhances overall accuracy.

Updating existing models with past data is more cost-effective than retraining from scratch.

Abstract

A large obstacle to deploying deep learning models in practice is the process of updating models post-deployment (ideally, frequently). Deep neural networks can cost many thousands of dollars to train. When new data comes in the pipeline, you can train a new model from scratch (randomly initialized weights) on all existing data. Instead, you can take an existing model and fine-tune (continue to train) it on new data. The former is costly and slow. The latter is cheap and fast, but catastrophic forgetting generally causes the new model to 'forget' how to classify older data well. There are a plethora of complicated techniques to keep models from forgetting their past learnings. Arguably the most basic is to mix in a small amount of past data into the new data during fine-tuning: also known as 'data rehearsal'. In this paper, we compare various methods of limiting catastrophic forgetting and conclude that if you can maintain access to a portion of your past data (or tasks), data rehearsal is ideal in terms of overall accuracy across all time periods, and performs even better when combined with methods like Elastic Weight Consolidation (EWC). Especially when the amount of past data (past 'tasks') is large compared to new data, the cost of updating an existing model is far cheaper and faster than training a new model from scratch.