Automatic Mapping of the Best-Suited DNN Pruning Schemes for Real-Time Mobile Acceleration

TL;DR

This paper introduces a flexible, fine-grained structured pruning method and compiler optimizations that automatically identify the best pruning schemes for each DNN layer, significantly improving inference speed on mobile devices without accuracy loss.

Contribution

It presents a novel, general pruning scheme applicable to all DNN layers and two automatic mapping methods to optimize pruning for hardware acceleration and accuracy.

Findings

Achieves up to 2.48× and 1.73× inference acceleration on CIFAR-10 and ImageNet.

Outperforms state-of-the-art DNN optimization frameworks.

Maintains accuracy while significantly improving inference speed.

Abstract

Weight pruning is an effective model compression technique to tackle the challenges of achieving real-time deep neural network (DNN) inference on mobile devices. However, prior pruning schemes have limited application scenarios due to accuracy degradation, difficulty in leveraging hardware acceleration, and/or restriction on certain types of DNN layers. In this paper, we propose a general, fine-grained structured pruning scheme and corresponding compiler optimizations that are applicable to any type of DNN layer while achieving high accuracy and hardware inference performance. With the flexibility of applying different pruning schemes to different layers enabled by our compiler optimizations, we further probe into the new problem of determining the best-suited pruning scheme considering the different acceleration and accuracy performance of various pruning schemes. Two pruning scheme mapping methods, one is search-based and the other is rule-based, are proposed to automatically derive the best-suited pruning regularity and block size for each layer of any given DNN. Experimental results demonstrate that our pruning scheme mapping methods, together with the general fine-grained structured pruning scheme, outperform the state-of-the-art DNN optimization framework with up to 2.48$\times$ and 1.73$\times$ DNN inference acceleration on CIFAR-10 and ImageNet dataset without accuracy loss.