Weight-dependent Gates for Network Pruning

TL;DR

This paper introduces weight-dependent gates (W-Gates) for network pruning that automatically decide which filters to prune based on weights, combined with an efficiency module to optimize for hardware constraints, resulting in more accurate and efficient models.

Contribution

The paper proposes a novel weight-dependent gating mechanism for automatic filter pruning and integrates an efficiency module for hardware-aware optimization, improving accuracy and efficiency trade-offs.

Findings

Achieved up to 1.33x higher Top-1 accuracy on ImageNet.

Reduced hardware latency while maintaining accuracy.

Outperformed state-of-the-art pruning methods.

Abstract

In this paper, a simple yet effective network pruning framework is proposed to simultaneously address the problems of pruning indicator, pruning ratio, and efficiency constraint. This paper argues that the pruning decision should depend on the convolutional weights, and thus proposes novel weight-dependent gates (W-Gates) to learn the information from filter weights and obtain binary gates to prune or keep the filters automatically. To prune the network under efficiency constraints, a switchable Efficiency Module is constructed to predict the hardware latency or FLOPs of candidate pruned networks. Combined with the proposed Efficiency Module, W-Gates can perform filter pruning in an efficiency-aware manner and achieve a compact network with a better accuracy-efficiency trade-off. We have demonstrated the effectiveness of the proposed method on ResNet34, ResNet50, and MobileNet V2, respectively achieving up to 1.33/1.28/1.1 higher Top-1 accuracy with lower hardware latency on ImageNet. Compared with state-of-the-art methods, W-Gates also achieves superior performance.