Juvenile state hypothesis: What we can learn from lottery ticket hypothesis researches?

TL;DR

This paper introduces a recursive network structure search combined with pruning to generate better lottery ticket sub-networks, addressing training costs and knowledge forgetting, and proposes a new hypothesis on factors maintaining neural network juvenility.

Contribution

It proposes a novel recursive structure search and pruning strategy to improve lottery ticket hypothesis outcomes and introduces a new hypothesis on neural network juvenility.

Findings

Enhanced sub-network performance with deeper structures.

Improved generalization ability of recursive lottery tickets.

Validated on MNIST and CIFAR-10 datasets.

Abstract

The proposition of lottery ticket hypothesis revealed the relationship between network structure and initialization parameters and the learning potential of neural networks. The original lottery ticket hypothesis performs pruning and weight resetting after training convergence, exposing it to the problem of forgotten learning knowledge and potential high cost of training. Therefore, we propose a strategy that combines the idea of neural network structure search with a pruning algorithm to alleviate this problem. This algorithm searches and extends the network structure on existing winning ticket sub-network to producing new winning ticket recursively. This allows the training and pruning process to continue without compromising performance. A new winning ticket sub-network with deeper network structure, better generalization ability and better test performance can be obtained in this recursive manner. This method can solve: the difficulty of training or performance degradation of the sub-networks after pruning, the forgetting of the weights of the original lottery ticket hypothesis and the difficulty of generating winning ticket sub-network when the final network structure is not given. We validate this strategy on the MNIST and CIFAR-10 datasets. And after relating it to similar biological phenomena and relevant lottery ticket hypothesis studies in recent years, we will further propose a new hypothesis to discuss which factors that can keep a network juvenile, i.e., those possible factors that influence the learning potential or generalization performance of a neural network during training.