Revisiting hard thresholding for DNN pruning

TL;DR

This paper compares hard thresholding and smart pruning for DNNs, showing hard thresholding remains most efficient overall, while proposing a faster smart pruning method with minimal accuracy loss and analyzing the theoretical effects of pruning.

Contribution

It introduces a novel fast smart pruning algorithm based on difference of convex functions optimization and provides theoretical insights into the impact of hard thresholding on DNN accuracy.

Findings

Hard thresholding with retraining is most efficient overall.

The proposed smart pruning method is significantly faster and maintains low accuracy degradation.

Accuracy loss increases with depth from the pruned layer and relates to data manifold dimensionality.

Abstract

The most common method for DNN pruning is hard thresholding of network weights, followed by retraining to recover any lost accuracy. Recently developed smart pruning algorithms use the DNN response over the training set for a variety of cost functions to determine redundant network weights, leading to less accuracy degradation and possibly less retraining time. For experiments on the total pruning time (pruning time + retraining time) we show that hard thresholding followed by retraining remains the most efficient way of reducing the number of network parameters. However smart pruning algorithms still have advantages when retraining is not possible. In this context we propose a novel smart pruning algorithm based on difference of convex functions optimisation and show that it is often orders of magnitude faster than competing approaches while achieving the lowest classification accuracy degradation. Furthermore we investigate theoretically the effect of hard thresholding on DNN accuracy. We show that accuracy degradation increases with remaining network depth from the pruned layer. We also discover a link between the latent dimensionality of the training data manifold and network robustness to hard thresholding.