Catalyst: a Novel Regularizer for Structured Pruning with Auxiliary Extension of Parameter Space

TL;DR

This paper introduces Catalyst, a new regularizer for structured neural network pruning that ensures fair, robust, and theoretically justified filter removal, leading to superior pruning performance across datasets.

Contribution

The paper proposes a novel regularizer in an extended parameter space that guarantees unbiased, wide-margin pruning decisions with theoretical validation.

Findings

Catalyst achieves superior pruning results compared to state-of-the-art methods.

The regularizer ensures unbiased, margin-based pruning decisions.

Empirical results validate the theoretical properties of Catalyst pruning.

Abstract

Structured pruning aims to reduce the size and computational cost of deep neural networks by removing entire filters or channels. The traditional regularizers such as L1 or Group Lasso and its variants lead to magnitude-biased pruning decisions, such that the filters with small magnitudes are likely to be pruned. Also, they often entail pruning results with almost zero margin around pruning decision boundary, such that tiny perturbation in a filter magnitude can flip the pruning decision. In this paper, we identify the precise algebraic condition under which pruning operations preserve model performance, and use the condition to construct a novel regularizer defined in an extended parameter space via auxiliary catalyst variables. The proposed Catalyst regularization ensures fair pruning chance for each filters with theoretically provable zero bias to their magnitude and robust pruning behavior achieved by wide-margin bifurcation of magnitudes between the preserved and the pruned filters. The theoretical properties naturally lead to real-world effectiveness, as shown by empirical validations of Catalyst Pruning algorithm. Pruning results on various datasets and models are superior to state-of-the-art filter pruning methods, and at the same time confirm the predicted robust and fair pruning characteristics of Catalyst pruning.