Fusion-Catalyzed Pruning for Optimizing Deep Learning on Intelligent Edge Devices

TL;DR

This paper introduces FuPruner, a novel fusion-catalyzed pruning method that optimizes both parametric and non-parametric operators in neural networks, significantly improving inference speed on resource-limited edge devices.

Contribution

FuPruner extends model optimization by enabling pruning of non-parametric operators through aggressive fusion, offering flexible performance-accuracy trade-offs for edge deployment.

Findings

Accelerates neural network inference on edge devices.

Maintains accuracy while reducing computational cost.

Effective on multiple neural network architectures and platforms.

Abstract

The increasing computational cost of deep neural network models limits the applicability of intelligent applications on resource-constrained edge devices. While a number of neural network pruning methods have been proposed to compress the models, prevailing approaches focus only on parametric operators (e.g., convolution), which may miss optimization opportunities. In this paper, we present a novel fusion-catalyzed pruning approach, called FuPruner, which simultaneously optimizes the parametric and non-parametric operators for accelerating neural networks. We introduce an aggressive fusion method to equivalently transform a model, which extends the optimization space of pruning and enables non-parametric operators to be pruned in a similar manner as parametric operators, and a dynamic filter pruning method is applied to decrease the computational cost of models while retaining the accuracy requirement. Moreover, FuPruner provides configurable optimization options for controlling fusion and pruning, allowing much more flexible performance-accuracy trade-offs to be made. Evaluation with state-of-the-art residual neural networks on five representative intelligent edge platforms, Jetson TX2, Jetson Nano, Edge TPU, NCS, and NCS2, demonstrates the effectiveness of our approach, which can accelerate the inference of models on CIFAR-10 and ImageNet datasets.