WoodFisher: Efficient Second-Order Approximation for Neural Network Compression

TL;DR

WoodFisher introduces an efficient second-order approximation method for neural network compression, significantly improving pruning performance and accuracy over existing techniques, and enabling automatic layer-wise thresholding.

Contribution

The paper proposes WoodFisher, a novel efficient inverse Hessian approximation method that enhances neural network pruning and compression, outperforming state-of-the-art approaches.

Findings

Outperforms popular pruning methods in one-shot scenarios

Achieves higher test accuracy with iterative pruning on ResNet-50 and MobileNetV1

Enables automatic layer-wise pruning threshold setting

Abstract

Second-order information, in the form of Hessian- or Inverse-Hessian-vector products, is a fundamental tool for solving optimization problems. Recently, there has been significant interest in utilizing this information in the context of deep neural networks; however, relatively little is known about the quality of existing approximations in this context. Our work examines this question, identifies issues with existing approaches, and proposes a method called WoodFisher to compute a faithful and efficient estimate of the inverse Hessian. Our main application is to neural network compression, where we build on the classic Optimal Brain Damage/Surgeon framework. We demonstrate that WoodFisher significantly outperforms popular state-of-the-art methods for one-shot pruning. Further, even when iterative, gradual pruning is considered, our method results in a gain in test accuracy over the state-of-the-art approaches, for pruning popular neural networks (like ResNet-50, MobileNetV1) trained on standard image classification datasets such as ImageNet ILSVRC. We examine how our method can be extended to take into account first-order information, as well as illustrate its ability to automatically set layer-wise pruning thresholds and perform compression in the limited-data regime. The code is available at the following link, https://github.com/IST-DASLab/WoodFisher.