Dynamic Sparse Training: Find Efficient Sparse Network From Scratch With Trainable Masked Layers

TL;DR

This paper introduces Dynamic Sparse Training, a novel method that jointly optimizes network weights and sparsity structure during training, achieving high performance with minimal loss and outperforming existing sparse training algorithms.

Contribution

It proposes a unified training approach with trainable pruning thresholds that dynamically adjust layer-wise sparsity, improving efficiency and performance over traditional pruning methods.

Findings

Achieves state-of-the-art sparse network performance.

Enables training of highly sparse models with little accuracy loss.

Provides insights into traditional pruning limitations.

Abstract

We present a novel network pruning algorithm called Dynamic Sparse Training that can jointly find the optimal network parameters and sparse network structure in a unified optimization process with trainable pruning thresholds. These thresholds can have fine-grained layer-wise adjustments dynamically via backpropagation. We demonstrate that our dynamic sparse training algorithm can easily train very sparse neural network models with little performance loss using the same number of training epochs as dense models. Dynamic Sparse Training achieves the state of the art performance compared with other sparse training algorithms on various network architectures. Additionally, we have several surprising observations that provide strong evidence for the effectiveness and efficiency of our algorithm. These observations reveal the underlying problems of traditional three-stage pruning algorithms and present the potential guidance provided by our algorithm to the design of more compact network architectures.