On the Sparsity of the Strong Lottery Ticket Hypothesis

TL;DR

This paper proves the Strong Lottery Ticket Hypothesis with guarantees on subnetwork sparsity, using new bounds on a subset sum problem, advancing understanding of neural network pruning without training.

Contribution

First proof of the SLTH with sparsity guarantees in classical neural network settings, employing tight bounds on a subset sum problem variant.

Findings

Established sparsity guarantees for subnetworks in dense and equivariant networks.

Derived tight bounds on the Random Fixed-Size Subset Sum Problem.

Enhanced theoretical understanding of neural network pruning and approximation.

Abstract

Considerable research efforts have recently been made to show that a random neural network $N$ contains subnetworks capable of accurately approximating any given neural network that is sufficiently smaller than $N$, without any training. This line of research, known as the Strong Lottery Ticket Hypothesis (SLTH), was originally motivated by the weaker Lottery Ticket Hypothesis, which states that a sufficiently large random neural network $N$ contains \emph{sparse} subnetworks that can be trained efficiently to achieve performance comparable to that of training the entire network $N$. Despite its original motivation, results on the SLTH have so far not provided any guarantee on the size of subnetworks. Such limitation is due to the nature of the main technical tool leveraged by these results, the Random Subset Sum (RSS) Problem. Informally, the RSS Problem asks how large a random i.i.d. sample $\Omega$ should be so that we are able to approximate any number in $[-1,1]$, up to an error of $ \epsilon$, as the sum of a suitable subset of $\Omega$. We provide the first proof of the SLTH in classical settings, such as dense and equivariant networks, with guarantees on the sparsity of the subnetworks. Central to our results, is the proof of an essentially tight bound on the Random Fixed-Size Subset Sum Problem (RFSS), a variant of the RSS Problem in which we only ask for subsets of a given size, which is of independent interest.