Efficient Latency-Aware CNN Depth Compression via Two-Stage Dynamic Programming

TL;DR

This paper introduces a two-stage dynamic programming approach for efficient CNN depth compression that merges convolution layers to reduce inference latency, outperforming existing methods in speed and accuracy.

Contribution

It proposes a novel subset selection formulation for depth compression, solved via dynamic programming, enabling faster and more accurate CNN inference.

Findings

Achieves 1.41x speed-up on MobileNetV2 with minimal accuracy loss

Outperforms baseline methods in inference speed and accuracy

Effective end-to-end latency reduction for convolutional neural networks

Abstract

Recent works on neural network pruning advocate that reducing the depth of the network is more effective in reducing run-time memory usage and accelerating inference latency than reducing the width of the network through channel pruning. In this regard, some recent works propose depth compression algorithms that merge convolution layers. However, the existing algorithms have a constricted search space and rely on human-engineered heuristics. In this paper, we propose a novel depth compression algorithm which targets general convolution operations. We propose a subset selection problem that replaces inefficient activation layers with identity functions and optimally merges consecutive convolution operations into shallow equivalent convolution operations for efficient end-to-end inference latency. Since the proposed subset selection problem is NP-hard, we formulate a surrogate optimization problem that can be solved exactly via two-stage dynamic programming within a few seconds. We evaluate our methods and baselines by TensorRT for a fair inference latency comparison. Our method outperforms the baseline method with higher accuracy and faster inference speed in MobileNetV2 on the ImageNet dataset. Specifically, we achieve $1.41\times$ speed-up with $0.11$\%p accuracy gain in MobileNetV2-1.0 on the ImageNet.