Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgetting
Xilai Li, Yingbo Zhou, Tianfu Wu, Richard Socher, Caiming Xiong

TL;DR
This paper introduces a simple, general framework for continual learning that separates neural structure optimization from parameter learning, effectively mitigating catastrophic forgetting in deep neural networks across multiple datasets.
Contribution
It proposes a novel framework that explicitly separates structure learning from parameter estimation, improving continual learning performance and overcoming catastrophic forgetting.
Findings
Outperforms baselines on permuted MNIST, split CIFAR100, and Visual Domain Decathlon datasets.
Effectively alleviates catastrophic forgetting in deep neural networks.
Demonstrates strong capabilities of evolving neural structures during continual learning.
Abstract
Addressing catastrophic forgetting is one of the key challenges in continual learning where machine learning systems are trained with sequential or streaming tasks. Despite recent remarkable progress in state-of-the-art deep learning, deep neural networks (DNNs) are still plagued with the catastrophic forgetting problem. This paper presents a conceptually simple yet general and effective framework for handling catastrophic forgetting in continual learning with DNNs. The proposed method consists of two components: a neural structure optimization component and a parameter learning and/or fine-tuning component. By separating the explicit neural structure learning and the parameter estimation, not only is the proposed method capable of evolving neural structures in an intuitively meaningful way, but also shows strong capabilities of alleviating catastrophic forgetting in experiments.…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsDomain Adaptation and Few-Shot Learning · Multimodal Machine Learning Applications · Advanced Neural Network Applications
