Plant 'n' Seek: Can You Find the Winning Ticket?

TL;DR

This paper introduces a framework to plant and hide winning lottery tickets in neural networks, enabling analysis of pruning algorithms' ability to identify sparse solutions and revealing current limitations of these methods.

Contribution

We develop a novel framework for planting and hiding winning tickets in neural networks, allowing systematic evaluation of pruning algorithms on tasks with ground truth.

Findings

Pruning algorithms can identify sparse tickets in our framework similar to real data.

Current pruning limitations are likely due to algorithms, not fundamental constraints.

Our framework provides transferable insights into pruning performance across tasks.

Abstract

The lottery ticket hypothesis has sparked the rapid development of pruning algorithms that aim to reduce the computational costs associated with deep learning during training and model deployment. Currently, such algorithms are primarily evaluated on imaging data, for which we lack ground truth information and thus the understanding of how sparse lottery tickets could be. To fill this gap, we develop a framework that allows us to plant and hide winning tickets with desirable properties in randomly initialized neural networks. To analyze the ability of state-of-the-art pruning to identify tickets of extreme sparsity, we design and hide such tickets solving four challenging tasks. In extensive experiments, we observe similar trends as in imaging studies, indicating that our framework can provide transferable insights into realistic problems. Additionally, we can now see beyond such relative trends and highlight limitations of current pruning methods. Based on our results, we conclude that the current limitations in ticket sparsity are likely of algorithmic rather than fundamental nature. We anticipate that comparisons to planted tickets will facilitate future developments of efficient pruning algorithms.