Juvenile state hypothesis: What we can learn from lottery ticket hypothesis researches?
Di Zhang

TL;DR
This paper introduces a recursive network structure search combined with pruning to generate better lottery ticket sub-networks, addressing training costs and knowledge forgetting, and proposes a new hypothesis on factors maintaining neural network juvenility.
Contribution
It proposes a novel recursive structure search and pruning strategy to improve lottery ticket hypothesis outcomes and introduces a new hypothesis on neural network juvenility.
Findings
Enhanced sub-network performance with deeper structures.
Improved generalization ability of recursive lottery tickets.
Validated on MNIST and CIFAR-10 datasets.
Abstract
The proposition of lottery ticket hypothesis revealed the relationship between network structure and initialization parameters and the learning potential of neural networks. The original lottery ticket hypothesis performs pruning and weight resetting after training convergence, exposing it to the problem of forgotten learning knowledge and potential high cost of training. Therefore, we propose a strategy that combines the idea of neural network structure search with a pruning algorithm to alleviate this problem. This algorithm searches and extends the network structure on existing winning ticket sub-network to producing new winning ticket recursively. This allows the training and pruning process to continue without compromising performance. A new winning ticket sub-network with deeper network structure, better generalization ability and better test performance can be obtained in this…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsNeural Networks and Applications · Domain Adaptation and Few-Shot Learning · Stochastic Gradient Optimization Techniques
MethodsPruning
