Towards Hardware-Specific Automatic Compression of Neural Networks

TL;DR

This paper introduces Galen, a reinforcement learning framework that automatically finds hardware-specific neural network compression policies, optimizing for inference latency on target devices while maintaining accuracy.

Contribution

Galen is the first framework to automatically optimize neural network compression policies considering hardware-specific latency, combining pruning and quantization via reinforcement learning.

Findings

Compressed ResNet18 to 20% latency on ARM with minimal accuracy loss.

Joint pruning and quantization outperform individual methods.

Reinforcement learning effectively finds hardware-aware compression policies.

Abstract

Compressing neural network architectures is important to allow the deployment of models to embedded or mobile devices, and pruning and quantization are the major approaches to compress neural networks nowadays. Both methods benefit when compression parameters are selected specifically for each layer. Finding good combinations of compression parameters, so-called compression policies, is hard as the problem spans an exponentially large search space. Effective compression policies consider the influence of the specific hardware architecture on the used compression methods. We propose an algorithmic framework called Galen to search such policies using reinforcement learning utilizing pruning and quantization, thus providing automatic compression for neural networks. Contrary to other approaches we use inference latency measured on the target hardware device as an optimization goal. With that, the framework supports the compression of models specific to a given hardware target. We validate our approach using three different reinforcement learning agents for pruning, quantization and joint pruning and quantization. Besides proving the functionality of our approach we were able to compress a ResNet18 for CIFAR-10, on an embedded ARM processor, to 20% of the original inference latency without significant loss of accuracy. Moreover, we can demonstrate that a joint search and compression using pruning and quantization is superior to an individual search for policies using a single compression method.