Demystifying Neural Network Filter Pruning

TL;DR

This paper analyzes neural network filter pruning from a functionality perspective, revealing limitations of magnitude-based methods and proposing a new approach that reduces redundancy without retraining.

Contribution

It introduces a functionality-oriented pruning method that effectively reduces filters' redundancy and maintains accuracy without retraining.

Findings

Magnitude-based pruning fails to remove redundant filters with similar functions.

Retraining primarily compensates for wrongly-pruned critical filters.

Functionality-focused pruning can eliminate redundancy without accuracy loss.

Abstract

Based on filter magnitude ranking (e.g. L1 norm), conventional filter pruning methods for Convolutional Neural Networks (CNNs) have been proved with great effectiveness in computation load reduction. Although effective, these methods are rarely analyzed in a perspective of filter functionality. In this work, we explore the filter pruning and the retraining through qualitative filter functionality interpretation. We find that the filter magnitude based method fails to eliminate the filters with repetitive functionality. And the retraining phase is actually used to reconstruct the remained filters for functionality compensation for the wrongly-pruned critical filters. With a proposed functionality-oriented pruning method, we further testify that, by precisely addressing the filter functionality redundancy, a CNN can be pruned without considerable accuracy drop, and the retraining phase is unnecessary.