One Shot vs. Iterative: Rethinking Pruning Strategies for Model Compression

TL;DR

This paper systematically compares one-shot and iterative pruning strategies for neural network compression, revealing their respective advantages at different pruning ratios and proposing a hybrid approach for improved performance.

Contribution

It provides the first comprehensive benchmarking and analysis of one-shot versus iterative pruning, including definitions, criteria, and practical recommendations.

Findings

One-shot pruning is more effective at low pruning ratios.

Iterative pruning outperforms at high pruning ratios.

A hybrid approach can surpass traditional methods in certain scenarios.

Abstract

Pruning is a core technique for compressing neural networks to improve computational efficiency. This process is typically approached in two ways: one-shot pruning, which involves a single pass of training and pruning, and iterative pruning, where pruning is performed over multiple cycles for potentially finer network refinement. Although iterative pruning has historically seen broader adoption, this preference is often assumed rather than rigorously tested. Our study presents one of the first systematic and comprehensive comparisons of these methods, providing rigorous definitions, benchmarking both across structured and unstructured settings, and applying different pruning criteria and modalities. We find that each method has specific advantages: one-shot pruning proves more effective at lower pruning ratios, while iterative pruning performs better at higher ratios. Building on these findings, we advocate for patience-based pruning and introduce a hybrid approach that can outperform traditional methods in certain scenarios, providing valuable insights for practitioners selecting a pruning strategy tailored to their goals and constraints. Source code is available at https://github.com/janumiko/pruning-benchmark.