DepGraph: Towards Any Structural Pruning
Gongfan Fang, Xinyin Ma, Mingli Song, Michael Bi Mi, Xinchao Wang

TL;DR
DepGraph introduces a universal, automatic approach for structural pruning across diverse neural network architectures by modeling layer dependencies to enable effective and general pruning without architecture-specific manual design.
Contribution
We propose DepGraph, a fully automatic dependency graph method that models layer dependencies for general structural pruning across various neural network architectures.
Findings
Consistently improves pruning performance across multiple architectures.
Effective even with simple norm-based criteria.
Demonstrates broad applicability to CNNs, RNNs, GNNs, and Transformers.
Abstract
Structural pruning enables model acceleration by removing structurally-grouped parameters from neural networks. However, the parameter-grouping patterns vary widely across different models, making architecture-specific pruners, which rely on manually-designed grouping schemes, non-generalizable to new architectures. In this work, we study a highly-challenging yet barely-explored task, any structural pruning, to tackle general structural pruning of arbitrary architecture like CNNs, RNNs, GNNs and Transformers. The most prominent obstacle towards this goal lies in the structural coupling, which not only forces different layers to be pruned simultaneously, but also expects all removed parameters to be consistently unimportant, thereby avoiding structural issues and significant performance degradation after pruning. To address this problem, we propose a general and {fully automatic} method,…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdvanced Neural Network Applications · Human Pose and Action Recognition · Multimodal Machine Learning Applications
MethodsAttention Is All You Need · Pruning · Batch Normalization · Concatenated Skip Connection · *Communicated@Fast*How Do I Communicate to Expedia? · Dense Block · 1x1 Convolution · Average Pooling · Max Pooling · Linear Layer
