Dynamic Guidance Adversarial Distillation with Enhanced Teacher Knowledge

TL;DR

This paper introduces DGAD, a novel adversarial distillation framework that dynamically guides knowledge transfer from a robust teacher to a student, improving accuracy and robustness against adversarial attacks.

Contribution

The paper proposes the DGAD framework with MAP, ELS, and PCR techniques to enhance adversarial distillation, addressing sample importance and misclassification issues.

Findings

DGAD improves clean data accuracy significantly.

DGAD enhances robustness against adversarial attacks.

Experimental results on CIFAR datasets validate effectiveness.

Abstract

In the realm of Adversarial Distillation (AD), strategic and precise knowledge transfer from an adversarially robust teacher model to a less robust student model is paramount. Our Dynamic Guidance Adversarial Distillation (DGAD) framework directly tackles the challenge of differential sample importance, with a keen focus on rectifying the teacher model's misclassifications. DGAD employs Misclassification-Aware Partitioning (MAP) to dynamically tailor the distillation focus, optimizing the learning process by steering towards the most reliable teacher predictions. Additionally, our Error-corrective Label Swapping (ELS) corrects misclassifications of the teacher on both clean and adversarially perturbed inputs, refining the quality of knowledge transfer. Further, Predictive Consistency Regularization (PCR) guarantees consistent performance of the student model across both clean and adversarial inputs, significantly enhancing its overall robustness. By integrating these methodologies, DGAD significantly improves upon the accuracy of clean data and fortifies the model's defenses against sophisticated adversarial threats. Our experimental validation on CIFAR10, CIFAR100, and Tiny ImageNet datasets, employing various model architectures, demonstrates the efficacy of DGAD, establishing it as a promising approach for enhancing both the robustness and accuracy of student models in adversarial settings.