How Sparse Can We Prune A Deep Network: A Fundamental Limit Perspective

TL;DR

This paper establishes the fundamental limit of deep network pruning by analyzing the phase transition point using convex geometry, revealing key factors like weight magnitude and sharpness that influence pruning ratios.

Contribution

It introduces a first-principles approach to determine the fundamental pruning limit, linking it to network properties and providing practical methods for estimation.

Findings

Theoretical pruning ratio threshold matches experimental results.

Identifies weight magnitude and sharpness as key factors affecting pruning limits.

Provides insights into heuristics used in existing pruning algorithms.

Abstract

Network pruning is a commonly used measure to alleviate the storage and computational burden of deep neural networks. However, the fundamental limit of network pruning is still lacking. To close the gap, in this work we'll take a first-principles approach, i.e. we'll directly impose the sparsity constraint on the loss function and leverage the framework of statistical dimension in convex geometry, thus enabling us to characterize the sharp phase transition point, which can be regarded as the fundamental limit of the pruning ratio. Through this limit, we're able to identify two key factors that determine the pruning ratio limit, namely, weight magnitude and network sharpness. Generally speaking, the flatter the loss landscape or the smaller the weight magnitude, the smaller pruning ratio. Moreover, we provide efficient countermeasures to address the challenges in the computation of the pruning limit, which mainly involves the accurate spectrum estimation of a large-scale and non-positive Hessian matrix. Moreover, through the lens of the pruning ratio threshold, we can also provide rigorous interpretations on several heuristics in existing pruning algorithms. Extensive experiments are performed which demonstrate that our theoretical pruning ratio threshold coincides very well with the experiments. All codes are available at: https://github.com/QiaozheZhang/Global-One-shot-Pruning