Structured Directional Pruning via Perturbation Orthogonal Projection

TL;DR

This paper introduces a novel structured directional pruning method that projects perturbations orthogonally onto the flat minimum valley, achieving state-of-the-art accuracy without retraining across various neural network architectures and datasets.

Contribution

It proposes a new structured directional pruning technique based on orthogonal projection and a fast solver, sDprun, with theoretical guarantees and superior empirical performance.

Findings

Achieves 93.97% accuracy on VGG16 CIFAR-10 without retraining

Reaches the same minimum valley as the optimizer in experiments

Demonstrates effectiveness across multiple architectures and datasets

Abstract

Structured pruning is an effective compression technique to reduce the computation of neural networks, which is usually achieved by adding perturbations to reduce network parameters at the cost of slightly increasing training loss. A more reasonable approach is to find a sparse minimizer along the flat minimum valley found by optimizers, i.e. stochastic gradient descent, which keeps the training loss constant. To achieve this goal, we propose the structured directional pruning based on orthogonal projecting the perturbations onto the flat minimum valley. We also propose a fast solver sDprun and further prove that it achieves directional pruning asymptotically after sufficient training. Experiments using VGG-Net and ResNet on CIFAR-10 and CIFAR-100 datasets show that our method obtains the state-of-the-art pruned accuracy (i.e. 93.97% on VGG16, CIFAR-10 task) without retraining. Experiments using DNN, VGG-Net and WRN28X10 on MNIST, CIFAR-10 and CIFAR-100 datasets demonstrate our method performs structured directional pruning, reaching the same minimum valley as the optimizer.