RED++ : Data-Free Pruning of Deep Neural Networks via Input Splitting and Output Merging

TL;DR

RED++ introduces a data-free pruning method for deep neural networks that exploits neuron weight redundancies through input splitting and output merging, improving efficiency without requiring training data.

Contribution

It presents a novel, architecture-agnostic, data-free pruning protocol with theoretical guarantees and practical effectiveness demonstrated on various neural network architectures.

Findings

Outperforms other data-free pruning methods

Competitive with data-driven pruning techniques

Provides theoretical guarantees on accuracy preservation

Abstract

Pruning Deep Neural Networks (DNNs) is a prominent field of study in the goal of inference runtime acceleration. In this paper, we introduce a novel data-free pruning protocol RED++. Only requiring a trained neural network, and not specific to DNN architecture, we exploit an adaptive data-free scalar hashing which exhibits redundancies among neuron weight values. We study the theoretical and empirical guarantees on the preservation of the accuracy from the hashing as well as the expected pruning ratio resulting from the exploitation of said redundancies. We propose a novel data-free pruning technique of DNN layers which removes the input-wise redundant operations. This algorithm is straightforward, parallelizable and offers novel perspective on DNN pruning by shifting the burden of large computation to efficient memory access and allocation. We provide theoretical guarantees on RED++ performance and empirically demonstrate its superiority over other data-free pruning methods and its competitiveness with data-driven ones on ResNets, MobileNets and EfficientNets.