Beyond One-Way Pruning: Bidirectional Pruning-Regrowth for Extreme Accuracy-Sparsity Tradeoff

TL;DR

This paper introduces a bidirectional pruning-regrowth strategy that enhances model compression by selectively regenerating connections, maintaining high accuracy even at extreme sparsity levels, thus enabling more efficient deployment on hardware.

Contribution

It proposes a novel pruning-regrowth method that overcomes accuracy degradation at high sparsity, improving compression ratios beyond traditional pruning techniques.

Findings

Effectively maintains accuracy at high sparsity levels.

Enables models to meet strict hardware size constraints.

Mitigates performance loss in extreme compression scenarios.

Abstract

As a widely adopted model compression technique, model pruning has demonstrated strong effectiveness across various architectures. However, we observe that when sparsity exceeds a certain threshold, both iterative and one-shot pruning methods lead to a steep decline in model performance. This rapid degradation limits the achievable compression ratio and prevents models from meeting the stringent size constraints required by certain hardware platforms, rendering them inoperable. To overcome this limitation, we propose a bidirectional pruning-regrowth strategy. Starting from an extremely compressed network that satisfies hardware constraints, the method selectively regenerates critical connections to recover lost performance, effectively mitigating the sharp accuracy drop commonly observed under high sparsity conditions.