Catastrophic Forgetting in the Context of Model Updates
Rich Harang, Hillary Sanders

TL;DR
This paper compares methods to mitigate catastrophic forgetting in model updates, highlighting that data rehearsal combined with techniques like EWC offers superior accuracy and efficiency when maintaining access to past data.
Contribution
It demonstrates that simple data rehearsal, especially when combined with EWC, effectively reduces forgetting and improves model update efficiency.
Findings
Data rehearsal outperforms other methods in preserving past knowledge.
Combining data rehearsal with EWC enhances overall accuracy.
Updating existing models with past data is more cost-effective than retraining from scratch.
Abstract
A large obstacle to deploying deep learning models in practice is the process of updating models post-deployment (ideally, frequently). Deep neural networks can cost many thousands of dollars to train. When new data comes in the pipeline, you can train a new model from scratch (randomly initialized weights) on all existing data. Instead, you can take an existing model and fine-tune (continue to train) it on new data. The former is costly and slow. The latter is cheap and fast, but catastrophic forgetting generally causes the new model to 'forget' how to classify older data well. There are a plethora of complicated techniques to keep models from forgetting their past learnings. Arguably the most basic is to mix in a small amount of past data into the new data during fine-tuning: also known as 'data rehearsal'. In this paper, we compare various methods of limiting catastrophic forgetting…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsExplainable Artificial Intelligence (XAI) · Machine Learning and Algorithms · Domain Adaptation and Few-Shot Learning
