Bayesian Optimization with Clustering and Rollback for CNN Auto Pruning

TL;DR

This paper introduces a clustering and rollback approach to enhance Bayesian optimization for CNN auto pruning, significantly improving convergence rates without increasing runtime, especially for deep networks.

Contribution

It proposes a novel clustering and rollback method to address the curse of dimensionality in Bayesian optimization for CNN pruning.

Findings

Improved convergence rate of BO in pruning deep CNNs

No increase in running time with the new method

Effective on ResNet and MobileNet models

Abstract

Pruning is an effective technique for convolutional neural networks (CNNs) model compression, but it is difficult to find the optimal pruning policy due to the large design space. To improve the usability of pruning, many auto pruning methods have been developed. Recently, Bayesian optimization (BO) has been considered to be a competitive algorithm for auto pruning due to its solid theoretical foundation and high sampling efficiency. However, BO suffers from the curse of dimensionality. The performance of BO deteriorates when pruning deep CNNs, since the dimension of the design spaces increase. We propose a novel clustering algorithm that reduces the dimension of the design space to speed up the searching process. Subsequently, a rollback algorithm is proposed to recover the high-dimensional design space so that higher pruning accuracy can be obtained. We validate our proposed method on ResNet, MobileNetV1, and MobileNetV2 models. Experiments show that the proposed method significantly improves the convergence rate of BO when pruning deep CNNs with no increase in running time. The source code is available at https://github.com/fanhanwei/BOCR.