Strong Lottery Ticket Hypothesis with $\varepsilon$--perturbation

TL;DR

This paper extends the strong Lottery Ticket Hypothesis by incorporating $\varepsilon$-perturbations, reducing over-parameterization needs and demonstrating that SGD-perturbed weights improve pruning performance.

Contribution

It generalizes the theoretical framework of the strong LTH to include perturbations, lowering the over-parameterization threshold and linking SGD weight changes to effective perturbations.

Findings

$\varepsilon$-perturbation reduces over-parameterization requirements.

SGD-perturbed weights outperform unperturbed in pruning tasks.

Theoretical extension of subset sum to neural network weights.

Abstract

The strong Lottery Ticket Hypothesis (LTH) claims the existence of a subnetwork in a sufficiently large, randomly initialized neural network that approximates some target neural network without the need of training. We extend the theoretical guarantee of the strong LTH literature to a scenario more similar to the original LTH, by generalizing the weight change in the pre-training step to some perturbation around initialization. In particular, we focus on the following open questions: By allowing an $\varepsilon$-scale perturbation on the random initial weights, can we reduce the over-parameterization requirement for the candidate network in the strong LTH? Furthermore, does the weight change by SGD coincide with a good set of such perturbation? We answer the first question by first extending the theoretical result on subset sum to allow perturbation on the candidates. Applying this result to the neural network setting, we show that such $\varepsilon$-perturbation reduces the over-parameterization requirement of the strong LTH. To answer the second question, we show via experiments that the perturbed weight achieved by the projected SGD shows better performance under the strong LTH pruning.