Network Morphism
Tao Wei, Changhu Wang, Yong Rui, Chang Wen Chen

TL;DR
This paper introduces a systematic method called network morphism that transforms trained neural networks into new architectures while preserving their functions, enabling knowledge inheritance and faster training for more powerful models.
Contribution
It develops a comprehensive framework and algorithms for network morphism that handle various network modifications and non-linearities, advancing neural network architecture evolution.
Findings
Effective network morphism schemes demonstrated on benchmark datasets.
Ability to morph networks with diverse structural changes.
Preservation of network function after morphing.
Abstract
We present in this paper a systematic study on how to morph a well-trained neural network to a new one so that its network function can be completely preserved. We define this as \emph{network morphism} in this research. After morphing a parent network, the child network is expected to inherit the knowledge from its parent network and also has the potential to continue growing into a more powerful one with much shortened training time. The first requirement for this network morphism is its ability to handle diverse morphing types of networks, including changes of depth, width, kernel size, and even subnet. To meet this requirement, we first introduce the network morphism equations, and then develop novel morphing algorithms for all these morphing types for both classic and convolutional neural networks. The second requirement for this network morphism is its ability to deal with…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsComplex Network Analysis Techniques
