[Reproducibility Report] Rigging the Lottery: Making All Tickets Winners

TL;DR

This paper reproduces and validates the performance of RigL, a sparse training algorithm, demonstrating its advantages over existing methods in training sparse neural networks efficiently and effectively.

Contribution

It provides a detailed reproduction of RigL, compares it with other methods, and explores hyperparameter tuning and initialization schemes for optimal sparse training.

Findings

RigL matches or exceeds existing dynamic-sparse training methods.

Training longer with RigL can surpass iterative pruning performance.

Hyperparameter choices are generally robust across sparsity levels.

Abstract

$\textit{RigL}$, a sparse training algorithm, claims to directly train sparse networks that match or exceed the performance of existing dense-to-sparse training techniques (such as pruning) for a fixed parameter count and compute budget. We implement $\textit{RigL}$ from scratch in Pytorch and reproduce its performance on CIFAR-10 within 0.1% of the reported value. On both CIFAR-10/100, the central claim holds -- given a fixed training budget, $\textit{RigL}$ surpasses existing dynamic-sparse training methods over a range of target sparsities. By training longer, the performance can match or exceed iterative pruning, while consuming constant FLOPs throughout training. We also show that there is little benefit in tuning $\textit{RigL}$'s hyper-parameters for every sparsity, initialization pair -- the reference choice of hyperparameters is often close to optimal performance. Going beyond the original paper, we find that the optimal initialization scheme depends on the training constraint. While the Erdos-Renyi-Kernel distribution outperforms the Uniform distribution for a fixed parameter count, for a fixed FLOP count, the latter performs better. Finally, redistributing layer-wise sparsity while training can bridge the performance gap between the two initialization schemes, but increases computational cost.