[Reproducibility Report] Rigging the Lottery: Making All Tickets Winners
Varun Sundar, Rajat Vadiraj Dwaraknath

TL;DR
This paper reproduces and validates the performance of RigL, a sparse training algorithm, demonstrating its advantages over existing methods in training sparse neural networks efficiently and effectively.
Contribution
It provides a detailed reproduction of RigL, compares it with other methods, and explores hyperparameter tuning and initialization schemes for optimal sparse training.
Findings
RigL matches or exceeds existing dynamic-sparse training methods.
Training longer with RigL can surpass iterative pruning performance.
Hyperparameter choices are generally robust across sparsity levels.
Abstract
, a sparse training algorithm, claims to directly train sparse networks that match or exceed the performance of existing dense-to-sparse training techniques (such as pruning) for a fixed parameter count and compute budget. We implement from scratch in Pytorch and reproduce its performance on CIFAR-10 within 0.1% of the reported value. On both CIFAR-10/100, the central claim holds -- given a fixed training budget, surpasses existing dynamic-sparse training methods over a range of target sparsities. By training longer, the performance can match or exceed iterative pruning, while consuming constant FLOPs throughout training. We also show that there is little benefit in tuning 's hyper-parameters for every sparsity, initialization pair -- the reference choice of hyperparameters is often close to optimal performance. Going beyond…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdvanced Neural Network Applications · Generative Adversarial Networks and Image Synthesis · Domain Adaptation and Few-Shot Learning
MethodsRigging the Lottery
