Signal Collapse in One-Shot Pruning: When Sparse Models Fail to Distinguish Neural Representations

TL;DR

This paper identifies signal collapse as the main cause of performance loss in one-shot pruning and introduces REFLOW, a method that mitigates this issue to significantly improve sparse model accuracy without weight updates.

Contribution

The paper proposes REFLOW, a novel approach that addresses signal collapse in pruning, enabling high-quality sparse networks without weight retraining, and achieves state-of-the-art results.

Findings

Restores ResNeXt101 accuracy from 4.1% to 78.9% on ImageNet with 20% weights

Outperforms existing pruning methods in accuracy

Highlights signal collapse as key to pruning performance

Abstract

Neural network pruning is essential for reducing model complexity to enable deployment on resource constrained hardware. While performance loss of pruned networks is often attributed to the removal of critical parameters, we identify signal collapse a reduction in activation variance across layers as the root cause. Existing one shot pruning methods focus on weight selection strategies and rely on computationally expensive second order approximations. In contrast, we demonstrate that mitigating signal collapse, rather than optimizing weight selection, is key to improving accuracy of pruned networks. We propose REFLOW that addresses signal collapse without updating trainable weights, revealing high quality sparse sub networks within the original parameter space. REFLOW enables magnitude pruning to achieve state of the art performance, restoring ResNeXt101 accuracy from under 4.1% to 78.9% on ImageNet with only 20% of the weights retained, surpassing state of the art approaches.