Not All Lotteries Are Made Equal
Surya Kant Sahu, Sai Mitheran, Somya Suhans Mahapatra

TL;DR
This paper explores how the size of neural networks affects the ability to find sparse sub-networks that perform comparably to dense models, revealing smaller models may be more advantageous under limited training resources.
Contribution
It demonstrates experimentally that smaller models are more effective for Ticket Search within finite training budgets, challenging assumptions about larger models' advantages.
Findings
Smaller models benefit more from Ticket Search under limited training budgets.
The relation between model size and ease of finding lottery tickets is inversely proportional.
Experimental evidence supports the efficiency of sparse sub-networks in smaller models.
Abstract
The Lottery Ticket Hypothesis (LTH) states that for a reasonably sized neural network, a sub-network within the same network yields no less performance than the dense counterpart when trained from the same initialization. This work investigates the relation between model size and the ease of finding these sparse sub-networks. We show through experiments that, surprisingly, under a finite budget, smaller models benefit more from Ticket Search (TS).
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMachine Learning and Data Classification · Machine Learning and Algorithms · Neural Networks and Applications
