Optimal Brain Connection: Towards Efficient Structural Pruning

TL;DR

This paper introduces a novel structural pruning framework called Optimal Brain Connection, which uses a Jacobian Criterion to evaluate parameter importance and an Equivalent Pruning mechanism with autoencoders to maintain performance during fine-tuning.

Contribution

It presents a new first-order metric for structural pruning that captures parameter interactions and dependencies, improving model compression effectiveness.

Findings

Jacobian Criterion outperforms existing metrics in preserving accuracy

Equivalent Pruning reduces performance loss after fine-tuning

Framework achieves efficient neural network compression

Abstract

Structural pruning has been widely studied for its effectiveness in compressing neural networks. However, existing methods often neglect the interconnections among parameters. To address this limitation, this paper proposes a structural pruning framework termed Optimal Brain Connection. First, we introduce the Jacobian Criterion, a first-order metric for evaluating the saliency of structural parameters. Unlike existing first-order methods that assess parameters in isolation, our criterion explicitly captures both intra-component interactions and inter-layer dependencies. Second, we propose the Equivalent Pruning mechanism, which utilizes autoencoders to retain the contributions of all original connection--including pruned ones--during fine-tuning. Experimental results demonstrate that the Jacobian Criterion outperforms several popular metrics in preserving model performance, while the Equivalent Pruning mechanism effectively mitigates performance degradation after fine-tuning. Code: https://github.com/ShaowuChen/Optimal_Brain_Connection