Improving Adversarial Robustness Through Adaptive Learning-Driven Multi-Teacher Knowledge Distillation

TL;DR

This paper introduces an adaptive multi-teacher knowledge distillation approach to improve CNNs' adversarial robustness by leveraging adversarially trained teachers and an adaptive weighting strategy, without exposing the student to adversarial data.

Contribution

The paper proposes a novel adaptive learning-driven multi-teacher knowledge distillation method that enhances adversarial robustness of CNNs without using adversarial data for training the student.

Findings

Enhanced robustness against various adversarial attacks.

Effective knowledge transfer from adversarially trained teachers.

Improved accuracy on clean data while resisting attacks.

Abstract

Convolutional neural networks (CNNs) excel in computer vision but are susceptible to adversarial attacks, crafted perturbations designed to mislead predictions. Despite advances in adversarial training, a gap persists between model accuracy and robustness. To mitigate this issue, in this paper, we present a multi-teacher adversarial robustness distillation using an adaptive learning strategy. Specifically, our proposed method first trained multiple clones of a baseline CNN model using an adversarial training strategy on a pool of perturbed data acquired through different adversarial attacks. Once trained, these adversarially trained models are used as teacher models to supervise the learning of a student model on clean data using multi-teacher knowledge distillation. To ensure an effective robustness distillation, we design an adaptive learning strategy that controls the knowledge contribution of each model by assigning weights as per their prediction precision. Distilling knowledge from adversarially pre-trained teacher models not only enhances the learning capabilities of the student model but also empowers it with the capacity to withstand different adversarial attacks, despite having no exposure to adversarial data. To verify our claims, we extensively evaluated our proposed method on MNIST-Digits and Fashion-MNIST datasets across diverse experimental settings. The obtained results exhibit the efficacy of our multi-teacher adversarial distillation and adaptive learning strategy, enhancing CNNs' adversarial robustness against various adversarial attacks.