Preserving Deep Representations In One-Shot Pruning: A Hessian-Free Second-Order Optimization Framework

TL;DR

SNOWS introduces a Hessian-free second-order optimization framework for one-shot network pruning, effectively preserving deep representations and improving inference efficiency without retraining.

Contribution

It proposes a global nonlinear reconstruction objective and a Hessian-free optimization method for better pruning of deep networks in a single shot.

Findings

Achieves state-of-the-art pruning results on residual networks and Vision Transformers.

Effectively exploits nonlinearities in deep feature representations.

Compatible with prior sparse masks for improved weight adjustment.

Abstract

We present SNOWS, a one-shot post-training pruning framework aimed at reducing the cost of vision network inference without retraining. Current leading one-shot pruning methods minimize layer-wise least squares reconstruction error which does not take into account deeper network representations. We propose to optimize a more global reconstruction objective. This objective accounts for nonlinear activations deep in the network to obtain a better proxy for the network loss. This nonlinear objective leads to a more challenging optimization problem -- we demonstrate it can be solved efficiently using a specialized second-order optimization framework. A key innovation of our framework is the use of Hessian-free optimization to compute exact Newton descent steps without needing to compute or store the full Hessian matrix. A distinct advantage of SNOWS is that it can be readily applied on top of any sparse mask derived from prior methods, readjusting their weights to exploit nonlinearities in deep feature representations. SNOWS obtains state-of-the-art results on various one-shot pruning benchmarks including residual networks and Vision Transformers (ViT/B-16 and ViT/L-16, 86m and 304m parameters respectively).