Can Unstructured Pruning Reduce the Depth in Deep Neural Networks?

TL;DR

This paper introduces EGP, an entropy-guided pruning algorithm that can fully remove layers in deep neural networks, significantly reducing size while maintaining performance, and offers new insights into the relationship between entropy and pruning.

Contribution

The paper presents a novel entropy-guided pruning method capable of removing entire layers, advancing network compression techniques beyond traditional pruning limits.

Findings

EGP effectively compresses models like ResNet-18 and Swin-T.

EGP maintains competitive performance after aggressive pruning.

Provides insights into entropy's role in pruning effectiveness.

Abstract

Pruning is a widely used technique for reducing the size of deep neural networks while maintaining their performance. However, such a technique, despite being able to massively compress deep models, is hardly able to remove entire layers from a model (even when structured): is this an addressable task? In this study, we introduce EGP, an innovative Entropy Guided Pruning algorithm aimed at reducing the size of deep neural networks while preserving their performance. The key focus of EGP is to prioritize pruning connections in layers with low entropy, ultimately leading to their complete removal. Through extensive experiments conducted on popular models like ResNet-18 and Swin-T, our findings demonstrate that EGP effectively compresses deep neural networks while maintaining competitive performance levels. Our results not only shed light on the underlying mechanism behind the advantages of unstructured pruning, but also pave the way for further investigations into the intricate relationship between entropy, pruning techniques, and deep learning performance. The EGP algorithm and its insights hold great promise for advancing the field of network compression and optimization. The source code for EGP is released open-source.