AutoPruning for Deep Neural Network with Dynamic Channel Masking

TL;DR

This paper introduces AutoPruning, a learning-based method for neural network pruning that dynamically determines channel configurations using a hyperparameter and an optimization process, improving efficiency without relying on manual rules.

Contribution

It presents a novel auto pruning algorithm inspired by AutoML, utilizing dynamic channel masking and a new loss function to balance accuracy and computational cost.

Findings

Achieves competitive pruning results on benchmark datasets.

Effectively balances model accuracy and pruning ratio.

Introduces a dynamic masking process for channel evolution.

Abstract

Modern deep neural network models are large and computationally intensive. One typical solution to this issue is model pruning. However, most current pruning algorithms depend on hand crafted rules or domain expertise. To overcome this problem, we propose a learning based auto pruning algorithm for deep neural network, which is inspired by recent automatic machine learning(AutoML). A two objectives' problem that aims for the the weights and the best channels for each layer is first formulated. An alternative optimization approach is then proposed to derive the optimal channel numbers and weights simultaneously. In the process of pruning, we utilize a searchable hyperparameter, remaining ratio, to denote the number of channels in each convolution layer, and then a dynamic masking process is proposed to describe the corresponding channel evolution. To control the trade-off between the accuracy of a model and the pruning ratio of floating point operations, a novel loss function is further introduced. Preliminary experimental results on benchmark datasets demonstrate that our scheme achieves competitive results for neural network pruning.