IDAP++: Advancing Divergence-Based Pruning via Filter-Level and Layer-Level Optimization

TL;DR

This paper introduces IDAP++, a divergence-based neural network pruning method that optimizes filters and layers by analyzing information flow divergence, leading to significant compression while maintaining accuracy across diverse architectures.

Contribution

The paper proposes a unified framework using tensor flow divergence for filter and layer-level pruning, adaptable to various neural network architectures, improving compression efficiency.

Findings

Achieves substantial parameter reduction across multiple architectures.

Maintains competitive accuracy after pruning.

Outperforms state-of-the-art methods in model compression.

Abstract

This paper presents a novel approach to neural network compression that addresses redundancy at both the filter and architectural levels through a unified framework grounded in information flow analysis. Building on the concept of tensor flow divergence, which quantifies how information is transformed across network layers, we develop a two-stage optimization process. The first stage employs iterative divergence-aware pruning to identify and remove redundant filters while preserving critical information pathways. The second stage extends this principle to higher-level architecture optimization by analyzing layer-wise contributions to information propagation and selectively eliminating entire layers that demonstrate minimal impact on network performance. The proposed method naturally adapts to diverse architectures, including convolutional networks, transformers, and hybrid designs, providing a consistent metric for comparing the structural importance across different layer types. Experimental validation across multiple modern architectures and datasets reveals that this combined approach achieves substantial model compression while maintaining competitive accuracy. The presented approach achieves parameter reduction results that are globally comparable to those of state-of-the-art solutions and outperforms them across a wide range of modern neural network architectures, from convolutional models to transformers. The results demonstrate how flow divergence serves as an effective guiding principle for both filter-level and layer-level optimization, offering practical benefits for deployment in resource-constrained environments.