Pruning and Quantization for Deep Neural Network Acceleration: A Survey

TL;DR

This survey reviews pruning and quantization techniques for deep neural network compression, highlighting their methods, trade-offs, and effectiveness in enabling efficient real-time deployment with minimal accuracy loss.

Contribution

It provides a comprehensive comparison of pruning and quantization methods, analyzing their strengths, weaknesses, and practical applications in neural network compression.

Findings

Pruning can be static or dynamic, with various criteria for redundancy removal.

Quantization typically reduces precision to 8-bit integers, with lower bit widths also explored.

Combined pruning and quantization can further optimize network efficiency.

Abstract

Deep neural networks have been applied in many applications exhibiting extraordinary abilities in the field of computer vision. However, complex network architectures challenge efficient real-time deployment and require significant computation resources and energy costs. These challenges can be overcome through optimizations such as network compression. Network compression can often be realized with little loss of accuracy. In some cases accuracy may even improve. This paper provides a survey on two types of network compression: pruning and quantization. Pruning can be categorized as static if it is performed offline or dynamic if it is performed at run-time. We compare pruning techniques and describe criteria used to remove redundant computations. We discuss trade-offs in element-wise, channel-wise, shape-wise, filter-wise, layer-wise and even network-wise pruning. Quantization reduces computations by reducing the precision of the datatype. Weights, biases, and activations may be quantized typically to 8-bit integers although lower bit width implementations are also discussed including binary neural networks. Both pruning and quantization can be used independently or combined. We compare current techniques, analyze their strengths and weaknesses, present compressed network accuracy results on a number of frameworks, and provide practical guidance for compressing networks.