Finding Dynamics Preserving Adversarial Winning Tickets

TL;DR

This paper introduces Adversarial Winning Tickets (AWT), sparse sub-networks that preserve adversarial training dynamics and match dense network robustness, verified through theoretical analysis and empirical experiments.

Contribution

It proves the existence of trainable sparse sub-networks for adversarial robustness at initialization, extending the lottery ticket hypothesis to adversarial training.

Findings

AWT preserves adversarial training dynamics

AWT achieves comparable robustness to dense networks

Theoretical verification of lottery tickets in adversarial context

Abstract

Modern deep neural networks (DNNs) are vulnerable to adversarial attacks and adversarial training has been shown to be a promising method for improving the adversarial robustness of DNNs. Pruning methods have been considered in adversarial context to reduce model capacity and improve adversarial robustness simultaneously in training. Existing adversarial pruning methods generally mimic the classical pruning methods for natural training, which follow the three-stage 'training-pruning-fine-tuning' pipelines. We observe that such pruning methods do not necessarily preserve the dynamics of dense networks, making it potentially hard to be fine-tuned to compensate the accuracy degradation in pruning. Based on recent works of \textit{Neural Tangent Kernel} (NTK), we systematically study the dynamics of adversarial training and prove the existence of trainable sparse sub-network at initialization which can be trained to be adversarial robust from scratch. This theoretically verifies the \textit{lottery ticket hypothesis} in adversarial context and we refer such sub-network structure as \textit{Adversarial Winning Ticket} (AWT). We also show empirical evidences that AWT preserves the dynamics of adversarial training and achieve equal performance as dense adversarial training.