Fantastic Weights and How to Find Them: Where to Prune in Dynamic Sparse Training

TL;DR

This paper investigates how different pruning criteria affect Dynamic Sparse Training, revealing that simple magnitude-based pruning performs best at low densities, with most methods yielding similar results otherwise.

Contribution

It provides an extensive empirical analysis of pruning criteria in DST, highlighting the effectiveness of simple magnitude-based pruning in low-density regimes.

Findings

Most pruning methods yield similar results in DST.

Magnitude-based pruning performs best at low densities.

Differences among methods are more pronounced at low sparsity levels.

Abstract

Dynamic Sparse Training (DST) is a rapidly evolving area of research that seeks to optimize the sparse initialization of a neural network by adapting its topology during training. It has been shown that under specific conditions, DST is able to outperform dense models. The key components of this framework are the pruning and growing criteria, which are repeatedly applied during the training process to adjust the network's sparse connectivity. While the growing criterion's impact on DST performance is relatively well studied, the influence of the pruning criterion remains overlooked. To address this issue, we design and perform an extensive empirical analysis of various pruning criteria to better understand their impact on the dynamics of DST solutions. Surprisingly, we find that most of the studied methods yield similar results. The differences become more significant in the low-density regime, where the best performance is predominantly given by the simplest technique: magnitude-based pruning. The code is provided at https://github.com/alooow/fantastic_weights_paper