Rigging the Lottery: Making All Tickets Winners
Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, Erich Elsen

TL;DR
This paper introduces a novel method for training sparse neural networks that maintains fixed parameter count and computational cost, improves accuracy, and reduces FLOPs by dynamically updating network topology during training.
Contribution
The paper presents a new topology-changing sparse training method that outperforms prior dense-to-sparse techniques in accuracy and efficiency across various models and datasets.
Findings
Achieves state-of-the-art sparse training results on multiple networks.
Requires fewer FLOPs to reach target accuracy.
Dynamic topology updates help overcome local minima.
Abstract
Many applications require sparse neural networks due to space or inference time restrictions. There is a large body of work on training dense networks to yield sparse networks for inference, but this limits the size of the largest trainable sparse model to that of the largest trainable dense model. In this paper we introduce a method to train sparse neural networks with a fixed parameter count and a fixed computational cost throughout training, without sacrificing accuracy relative to existing dense-to-sparse training methods. Our method updates the topology of the sparse network during training by using parameter magnitudes and infrequent gradient calculations. We show that this approach requires fewer floating-point operations (FLOPs) to achieve a given level of accuracy compared to prior techniques. We demonstrate state-of-the-art sparse training results on a variety of networks and…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
Taxonomy
TopicsSports Analytics and Performance · Artificial Intelligence in Games
MethodsRigging the Lottery
