Not All Lotteries Are Made Equal

TL;DR

This paper explores how the size of neural networks affects the ability to find sparse sub-networks that perform comparably to dense models, revealing smaller models may be more advantageous under limited training resources.

Contribution

It demonstrates experimentally that smaller models are more effective for Ticket Search within finite training budgets, challenging assumptions about larger models' advantages.

Findings

Smaller models benefit more from Ticket Search under limited training budgets.

The relation between model size and ease of finding lottery tickets is inversely proportional.

Experimental evidence supports the efficiency of sparse sub-networks in smaller models.

Abstract

The Lottery Ticket Hypothesis (LTH) states that for a reasonably sized neural network, a sub-network within the same network yields no less performance than the dense counterpart when trained from the same initialization. This work investigates the relation between model size and the ease of finding these sparse sub-networks. We show through experiments that, surprisingly, under a finite budget, smaller models benefit more from Ticket Search (TS).