A Survey of Lottery Ticket Hypothesis

TL;DR

This survey comprehensively reviews the Lottery Ticket Hypothesis, highlighting its empirical and theoretical foundations, current challenges, and future research directions in neural network sparsity.

Contribution

It provides an in-depth analysis of existing LTH research, discusses open issues, and proposes a platform for standardized experiments and comparisons.

Findings

LTH demonstrates that sparse subnetworks can outperform dense models.

Current challenges include efficiency, scalability, and lack of standardized frameworks.

The survey identifies future research directions in LTH.

Abstract

The Lottery Ticket Hypothesis (LTH) states that a dense neural network model contains a highly sparse subnetwork (i.e., winning tickets) that can achieve even better performance than the original model when trained in isolation. While LTH has been proved both empirically and theoretically in many works, there still are some open issues, such as efficiency and scalability, to be addressed. Also, the lack of open-source frameworks and consensual experimental setting poses a challenge to future research on LTH. We, for the first time, examine previous research and studies on LTH from different perspectives. We also discuss issues in existing works and list potential directions for further exploration. This survey aims to provide an in-depth look at the state of LTH and develop a duly maintained platform to conduct experiments and compare with the most updated baselines.