Information Consistent Pruning: How to Efficiently Search for Sparse Networks?

TL;DR

This paper introduces Information Consistent Pruning (InfCoP), a novel method that reduces training time in iterative magnitude pruning of neural networks by using a flow-based stopping criterion, maintaining performance efficiently.

Contribution

The paper proposes a new stopping criterion for IMP that minimizes training time and eliminates retraining during intermediate steps, with theoretical insights and practical efficiency gains.

Findings

InfCoP reduces overall training time compared to existing IMP methods.

It maintains network performance without retraining during intermediate pruning steps.

Theoretical analysis supports the flow-based pruning approach.

Abstract

Iterative magnitude pruning methods (IMPs), proven to be successful in reducing the number of insignificant nodes in over-parameterized deep neural networks (DNNs), have been getting an enormous amount of attention with the rapid deployment of DNNs into cutting-edge technologies with computation and memory constraints. Despite IMPs popularity in pruning networks, a fundamental limitation of existing IMP algorithms is the significant training time required for each pruning iteration. Our paper introduces a novel \textit{stopping criterion} for IMPs that monitors information and gradient flows between networks layers and minimizes the training time. Information Consistent Pruning (\ourmethod{}) eliminates the need to retrain the network to its original performance during intermediate steps while maintaining overall performance at the end of the pruning process. Through our experiments, we demonstrate that our algorithm is more efficient than current IMPs across multiple dataset-DNN combinations. We also provide theoretical insights into the core idea of our algorithm alongside mathematical explanations of flow-based IMP. Our code is available at \url{https://github.com/Sekeh-Lab/InfCoP}.