RED : Looking for Redundancies for Data-Free Structured Compression of Deep Neural Networks

TL;DR

RED introduces a data-free, structured pruning method for deep neural networks that merges similar neurons and employs a novel separation technique, achieving performance comparable to data-driven approaches.

Contribution

It presents a novel data-free structured pruning approach using adaptive hashing, neuron merging, and uneven depthwise separation, advancing the field of model compression.

Findings

Red outperforms other data-free pruning methods.

Red achieves results similar to data-driven methods.

Red effectively reduces model complexity without data.

Abstract

Deep Neural Networks (DNNs) are ubiquitous in today's computer vision land-scape, despite involving considerable computational costs. The mainstream approaches for runtime acceleration consist in pruning connections (unstructured pruning) or, better, filters (structured pruning), both often requiring data to re-train the model. In this paper, we present RED, a data-free structured, unified approach to tackle structured pruning. First, we propose a novel adaptive hashing of the scalar DNN weight distribution densities to increase the number of identical neurons represented by their weight vectors. Second, we prune the network by merging redundant neurons based on their relative similarities, as defined by their distance. Third, we propose a novel uneven depthwise separation technique to further prune convolutional layers. We demonstrate through a large variety of benchmarks that RED largely outperforms other data-free pruning methods, often reaching performance similar to unconstrained, data-driven methods.