Adversarial Contrastive Distillation with Adaptive Denoising

TL;DR

This paper introduces CRDND, a novel adversarial distillation method that enhances small model robustness by modeling teacher instability and leveraging relationships among examples, achieving state-of-the-art results.

Contribution

The paper proposes a structured adversarial distillation approach with an adaptive module and contrastive relationships to improve robustness transfer.

Findings

CRDND outperforms previous methods on multiple attack benchmarks.

It effectively models teacher instability to enhance robustness.

The method achieves state-of-the-art robustness performance.

Abstract

Adversarial Robustness Distillation (ARD) is a novel method to boost the robustness of small models. Unlike general adversarial training, its robust knowledge transfer can be less easily restricted by the model capacity. However, the teacher model that provides the robustness of knowledge does not always make correct predictions, interfering with the student's robust performances. Besides, in the previous ARD methods, the robustness comes entirely from one-to-one imitation, ignoring the relationship between examples. To this end, we propose a novel structured ARD method called Contrastive Relationship DeNoise Distillation (CRDND). We design an adaptive compensation module to model the instability of the teacher. Moreover, we utilize the contrastive relationship to explore implicit robustness knowledge among multiple examples. Experimental results on multiple attack benchmarks show CRDND can transfer robust knowledge efficiently and achieves state-of-the-art performances.