A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendations

TL;DR

This survey comprehensively reviews deep neural network pruning techniques, categorizing methods, comparing different settings, and exploring emerging topics to guide future research and practical deployment in resource-constrained environments.

Contribution

It provides an up-to-date taxonomy, comparative analysis, and recommendations for neural network pruning, including a curated dataset and evaluation collection for future research.

Findings

Pruning methods vary across speedup, timing, technique, and fusion categories.

Emerging topics include pruning for large language and multimodal models.

The survey offers practical guidelines and a resource repository for researchers.

Abstract

Modern deep neural networks, particularly recent large language models, come with massive model sizes that require significant computational and storage resources. To enable the deployment of modern models on resource-constrained environments and accelerate inference time, researchers have increasingly explored pruning techniques as a popular research direction in neural network compression. However, there is a dearth of up-to-date comprehensive review papers on pruning. To address this issue, in this survey, we provide a comprehensive review of existing research works on deep neural network pruning in a taxonomy of 1) universal/specific speedup, 2) when to prune, 3) how to prune, and 4) fusion of pruning and other compression techniques. We then provide a thorough comparative analysis of eight pairs of contrast settings for pruning and explore emerging topics, including pruning for large language models, large multimodal models, post-training pruning, and different supervision levels for pruning to shed light on the commonalities and differences of existing methods and lay the foundation for further method development. To facilitate future research, we build a curated collection of datasets, networks, and evaluations on different applications. Finally, we provide valuable recommendations on selecting pruning methods and prospect several promising research directions. We build a repository at https://github.com/hrcheng1066/awesome-pruning.