Holistic Adversarially Robust Pruning

TL;DR

HARP is a novel holistic pruning method that significantly compresses neural networks by 99% while preserving accuracy and adversarial robustness, using a dynamic, layer-specific compression strategy.

Contribution

HARP introduces a global, adaptive pruning approach that balances compression, accuracy, and robustness through incremental regularization and layer-wise optimization.

Findings

Maintains accuracy and robustness at 99% compression.

Layer-specific non-uniform compression improves results.

Dynamic regularization effectively balances multiple objectives.

Abstract

Neural networks can be drastically shrunk in size by removing redundant parameters. While crucial for the deployment on resource-constraint hardware, oftentimes, compression comes with a severe drop in accuracy and lack of adversarial robustness. Despite recent advances, counteracting both aspects has only succeeded for moderate compression rates so far. We propose a novel method, HARP, that copes with aggressive pruning significantly better than prior work. For this, we consider the network holistically. We learn a global compression strategy that optimizes how many parameters (compression rate) and which parameters (scoring connections) to prune specific to each layer individually. Our method fine-tunes an existing model with dynamic regularization, that follows a step-wise incremental function balancing the different objectives. It starts by favoring robustness before shifting focus on reaching the target compression rate and only then handles the objectives equally. The learned compression strategies allow us to maintain the pre-trained model natural accuracy and its adversarial robustness for a reduction by 99% of the network original size. Moreover, we observe a crucial influence of non-uniform compression across layers.