Adversarial Training via Adaptive Knowledge Amalgamation of an Ensemble of Teachers

TL;DR

This paper proposes AT-AKA, a novel adversarial training method that uses an ensemble of teachers and adaptive knowledge amalgamation to improve robustness and generalization of deep neural networks against various adversarial attacks.

Contribution

It introduces a new adversarial training framework that combines ensemble teacher models with adaptive knowledge amalgamation to enhance robustness and generalization in DNNs.

Findings

AT-AKA outperforms existing adversarial training methods.

It achieves higher robustness against AutoAttack and other strong adversaries.

The method improves generalization of adversarial samples during training.

Abstract

Adversarial training (AT) is a popular method for training robust deep neural networks (DNNs) against adversarial attacks. Yet, AT suffers from two shortcomings: (i) the robustness of DNNs trained by AT is highly intertwined with the size of the DNNs, posing challenges in achieving robustness in smaller models; and (ii) the adversarial samples employed during the AT process exhibit poor generalization, leaving DNNs vulnerable to unforeseen attack types. To address these dual challenges, this paper introduces adversarial training via adaptive knowledge amalgamation of an ensemble of teachers (AT-AKA). In particular, we generate a diverse set of adversarial samples as the inputs to an ensemble of teachers; and then, we adaptively amalgamate the logtis of these teachers to train a generalized-robust student. Through comprehensive experiments, we illustrate the superior efficacy of AT-AKA over existing AT methods and adversarial robustness distillation techniques against cutting-edge attacks, including AutoAttack.