The State of Sparsity in Deep Neural Networks

TL;DR

This paper evaluates various sparsity techniques in deep neural networks, revealing that simple pruning methods often outperform complex ones and emphasizing the importance of large-scale benchmarks for model compression research.

Contribution

It provides a comprehensive large-scale comparison of sparsity methods, demonstrating the limitations of complex techniques and establishing rigorous baselines for future research.

Findings

Simple magnitude pruning matches or exceeds complex methods in performance.

Unstructured sparse models trained via pruning cannot be trained from scratch to match pruned models.

Large-scale benchmarks are essential for advancing model compression techniques.

Abstract

We rigorously evaluate three state-of-the-art techniques for inducing sparsity in deep neural networks on two large-scale learning tasks: Transformer trained on WMT 2014 English-to-German, and ResNet-50 trained on ImageNet. Across thousands of experiments, we demonstrate that complex techniques (Molchanov et al., 2017; Louizos et al., 2017b) shown to yield high compression rates on smaller datasets perform inconsistently, and that simple magnitude pruning approaches achieve comparable or better results. Additionally, we replicate the experiments performed by (Frankle & Carbin, 2018) and (Liu et al., 2018) at scale and show that unstructured sparse architectures learned through pruning cannot be trained from scratch to the same test set performance as a model trained with joint sparsification and optimization. Together, these results highlight the need for large-scale benchmarks in the field of model compression. We open-source our code, top performing model checkpoints, and results of all hyperparameter configurations to establish rigorous baselines for future work on compression and sparsification.