ThinResNet: A New Baseline for Structured Convolutional Networks Pruning

TL;DR

This paper evaluates recent neural network pruning methods against modern training practices and introduces ThinResNet as a new strong baseline for structured pruning, highlighting the importance of updated benchmarks.

Contribution

The study demonstrates that trivial model scaling outperforms recent pruning methods under current training practices and proposes ThinResNet as a new standard baseline for structured pruning.

Findings

Trivial scaling surpasses recent pruning results with modern training.

Updated benchmarks reveal the need to re-evaluate classical pruning methods.

ThinResNet provides a more challenging and relevant baseline for future pruning research.

Abstract

Pruning is a compression method which aims to improve the efficiency of neural networks by reducing their number of parameters while maintaining a good performance, thus enhancing the performance-to-cost ratio in nontrivial ways. Of particular interest are structured pruning techniques, in which whole portions of parameters are removed altogether, resulting in easier to leverage shrunk architectures. Since its growth in popularity in the recent years, pruning gave birth to countless papers and contributions, resulting first in critical inconsistencies in the way results are compared, and then to a collective effort to establish standardized benchmarks. However, said benchmarks are based on training practices that date from several years ago and do not align with current practices. In this work, we verify how results in the recent literature of pruning hold up against networks that underwent both state-of-the-art training methods and trivial model scaling. We find that the latter clearly and utterly outperform all the literature we compared to, proving that updating standard pruning benchmarks and re-evaluating classical methods in their light is an absolute necessity. We thus introduce a new challenging baseline to compare structured pruning to: ThinResNet.