Max-Affine Spline Insights Into Deep Network Pruning

TL;DR

This paper offers a theoretical framework using Max-Affine Splines to understand and improve deep network pruning, revealing insights into decision boundaries and interpretability, and achieving competitive pruning performance.

Contribution

It introduces a novel theoretical approach based on CPA DNs to analyze pruning effects, interpret existing methods, and develop a principled pruning strategy.

Findings

Theoretical analysis explains the decision boundary changes due to pruning.

Spline-based pruning criteria match or outperform state-of-the-art methods.

Extensive experiments support the proposed insights and strategies.

Abstract

In this paper, we study the importance of pruning in Deep Networks (DNs) and the yin & yang relationship between (1) pruning highly overparametrized DNs that have been trained from random initialization and (2) training small DNs that have been "cleverly" initialized. As in most cases practitioners can only resort to random initialization, there is a strong need to develop a grounded understanding of DN pruning. Current literature remains largely empirical, lacking a theoretical understanding of how pruning affects DNs' decision boundary, how to interpret pruning, and how to design corresponding principled pruning techniques. To tackle those questions, we propose to employ recent advances in the theoretical analysis of Continuous Piecewise Affine (CPA) DNs. From this perspective, we will be able to detect the early-bird (EB) ticket phenomenon, provide interpretability into current pruning techniques, and develop a principled pruning strategy. In each step of our study, we conduct extensive experiments supporting our claims and results; while our main goal is to enhance the current understanding towards DN pruning instead of developing a new pruning method, our spline pruning criteria in terms of layerwise and global pruning is on par with or even outperforms state-of-the-art pruning methods.