Layer-adaptive sparsity for the Magnitude-based Pruning

TL;DR

This paper introduces LAMP, a layer-adaptive magnitude-based pruning method that automatically determines layerwise sparsity without hyperparameter tuning, outperforming existing heuristics in neural network pruning tasks.

Contribution

The paper proposes a novel importance score, LAMP, for global pruning that adapts layerwise sparsity based on model-level distortion, eliminating the need for hyperparameter tuning.

Findings

LAMP outperforms existing layerwise sparsity selection methods across various image classification tasks.

LAMP maintains superior performance even in weight-rewinding setups.

Connectivity-oriented sparsity performs worse than simple global magnitude pruning in certain scenarios.

Abstract

Recent discoveries on neural network pruning reveal that, with a carefully chosen layerwise sparsity, a simple magnitude-based pruning achieves state-of-the-art tradeoff between sparsity and performance. However, without a clear consensus on "how to choose," the layerwise sparsities are mostly selected algorithm-by-algorithm, often resorting to handcrafted heuristics or an extensive hyperparameter search. To fill this gap, we propose a novel importance score for global pruning, coined layer-adaptive magnitude-based pruning (LAMP) score; the score is a rescaled version of weight magnitude that incorporates the model-level $\ell_2$ distortion incurred by pruning, and does not require any hyperparameter tuning or heavy computation. Under various image classification setups, LAMP consistently outperforms popular existing schemes for layerwise sparsity selection. Furthermore, we observe that LAMP continues to outperform baselines even in weight-rewinding setups, while the connectivity-oriented layerwise sparsity (the strongest baseline overall) performs worse than a simple global magnitude-based pruning in this case. Code: https://github.com/jaeho-lee/layer-adaptive-sparsity