AdaGAT: Adaptive Guidance Adversarial Training for the Robustness of Deep Neural Networks

TL;DR

AdaGAT introduces an adaptive guidance adversarial training method that dynamically adjusts the guide model to improve the robustness of deep neural networks against adversarial attacks.

Contribution

The paper proposes a novel AdaGAT method with dynamic guide model adjustment and dual loss functions for enhanced adversarial robustness transfer.

Findings

AdaGAT improves target model robustness across multiple datasets.

Dynamic guide adjustment outperforms static guidance methods.

The approach is effective against various adversarial attacks.

Abstract

Adversarial distillation (AD) is a knowledge distillation technique that facilitates the transfer of robustness from teacher deep neural network (DNN) models to lightweight target (student) DNN models, enabling the target models to perform better than only training the student model independently. Some previous works focus on using a small, learnable teacher (guide) model to improve the robustness of a student model. Since a learnable guide model starts learning from scratch, maintaining its optimal state for effective knowledge transfer during co-training is challenging. Therefore, we propose a novel Adaptive Guidance Adversarial Training (AdaGAT) method. Our method, AdaGAT, dynamically adjusts the training state of the guide model to install robustness to the target model. Specifically, we develop two separate loss functions as part of the AdaGAT method, allowing the guide model to participate more actively in backpropagation to achieve its optimal state. We evaluated our approach via extensive experiments on three datasets: CIFAR-10, CIFAR-100, and TinyImageNet, using the WideResNet-34-10 model as the target model. Our observations reveal that appropriately adjusting the guide model within a certain accuracy range enhances the target model's robustness across various adversarial attacks compared to a variety of baseline models.