Winning the Lottery with Continuous Sparsification

TL;DR

This paper introduces Continuous Sparsification, a novel method for pruning neural networks and finding lottery tickets, outperforming existing techniques in efficiency and accuracy across multiple models and datasets.

Contribution

It develops a new approximation of $ ext{l}_0$ regularization for pruning and lottery ticket search, surpassing state-of-the-art methods in performance and speed.

Findings

Outperforms existing pruning and ticket search methods

Achieves higher accuracy on CIFAR-10 and ImageNet

Enables fast parallel lottery ticket discovery

Abstract

The search for efficient, sparse deep neural network models is most prominently performed by pruning: training a dense, overparameterized network and removing parameters, usually via following a manually-crafted heuristic. Additionally, the recent Lottery Ticket Hypothesis conjectures that, for a typically-sized neural network, it is possible to find small sub-networks which, when trained from scratch on a comparable budget, match the performance of the original dense counterpart. We revisit fundamental aspects of pruning algorithms, pointing out missing ingredients in previous approaches, and develop a method, Continuous Sparsification, which searches for sparse networks based on a novel approximation of an intractable $\ell_0$ regularization. We compare against dominant heuristic-based methods on pruning as well as ticket search -- finding sparse subnetworks that can be successfully re-trained from an early iterate. Empirical results show that we surpass the state-of-the-art for both objectives, across models and datasets, including VGG trained on CIFAR-10 and ResNet-50 trained on ImageNet. In addition to setting a new standard for pruning, Continuous Sparsification also offers fast parallel ticket search, opening doors to new applications of the Lottery Ticket Hypothesis.