Pruning Adversarially Robust Neural Networks without Adversarial Examples

TL;DR

This paper introduces a novel adversarial pruning framework that maintains neural network robustness without generating adversarial examples, improving efficiency and adaptability across multiple datasets and attack types.

Contribution

It proposes a new method combining self-distillation and Hilbert-Schmidt Information Bottleneck to prune robust networks without needing adversarial examples during pruning.

Findings

Outperforms existing methods in robustness and efficiency

Effective across MNIST, CIFAR-10, and CIFAR-100 datasets

Resilient against five state-of-the-art attacks

Abstract

Adversarial pruning compresses models while preserving robustness. Current methods require access to adversarial examples during pruning. This significantly hampers training efficiency. Moreover, as new adversarial attacks and training methods develop at a rapid rate, adversarial pruning methods need to be modified accordingly to keep up. In this work, we propose a novel framework to prune a previously trained robust neural network while maintaining adversarial robustness, without further generating adversarial examples. We leverage concurrent self-distillation and pruning to preserve knowledge in the original model as well as regularizing the pruned model via the Hilbert-Schmidt Information Bottleneck. We comprehensively evaluate our proposed framework and show its superior performance in terms of both adversarial robustness and efficiency when pruning architectures trained on the MNIST, CIFAR-10, and CIFAR-100 datasets against five state-of-the-art attacks. Code is available at https://github.com/neu-spiral/PwoA/.