Proving the Lottery Ticket Hypothesis: Pruning is All You Need

TL;DR

This paper proves that large, randomly-initialized neural networks inherently contain smaller subnetworks capable of achieving target performance without additional training, strengthening the lottery ticket hypothesis.

Contribution

It establishes a formal proof that over-parameterized networks contain trainable subnetworks matching target accuracy without training, under broad conditions.

Findings

Subnetwork existence is guaranteed in over-parameterized networks.

No training is needed to achieve target performance with these subnetworks.

The proof applies to networks with bounded weights and distributions.

Abstract

The lottery ticket hypothesis (Frankle and Carbin, 2018), states that a randomly-initialized network contains a small subnetwork such that, when trained in isolation, can compete with the performance of the original network. We prove an even stronger hypothesis (as was also conjectured in Ramanujan et al., 2019), showing that for every bounded distribution and every target network with bounded weights, a sufficiently over-parameterized neural network with random weights contains a subnetwork with roughly the same accuracy as the target network, without any further training.