Deep Neural Networks pruning via the Structured Perspective Regularization

TL;DR

This paper introduces a novel structured pruning method for deep neural networks using a regularization technique derived from a mixed-integer programming formulation, leading to efficient model compression with competitive results.

Contribution

It proposes the Structured Perspective Regularization, a new regularization approach inspired by operational research, for effective structured pruning of neural networks.

Findings

Achieves competitive performance on ResNet architectures.

Effective in pruning models for CIFAR-10, CIFAR-100, and ImageNet.

Outperforms some existing structured pruning methods.

Abstract

In Machine Learning, Artificial Neural Networks (ANNs) are a very powerful tool, broadly used in many applications. Often, the selected (deep) architectures include many layers, and therefore a large amount of parameters, which makes training, storage and inference expensive. This motivated a stream of research about compressing the original networks into smaller ones without excessively sacrificing performances. Among the many proposed compression approaches, one of the most popular is \emph{pruning}, whereby entire elements of the ANN (links, nodes, channels, \ldots) and the corresponding weights are deleted. Since the nature of the problem is inherently combinatorial (what elements to prune and what not), we propose a new pruning method based on Operational Research tools. We start from a natural Mixed-Integer-Programming model for the problem, and we use the Perspective Reformulation technique to strengthen its continuous relaxation. Projecting away the indicator variables from this reformulation yields a new regularization term, which we call the Structured Perspective Regularization, that leads to structured pruning of the initial architecture. We test our method on some ResNet architectures applied to CIFAR-10, CIFAR-100 and ImageNet datasets, obtaining competitive performances w.r.t.~the state of the art for structured pruning.