Sample-wise Adaptive Weighting for Transfer Consistency in Adversarial Distillation

TL;DR

This paper introduces SAAD, a novel method that adaptively reweights training samples based on their transferability to improve adversarial robustness transfer from teacher to student networks.

Contribution

The paper proposes Sample-wise Adaptive Adversarial Distillation (SAAD), a new approach that enhances robustness transfer by reweighting examples according to their transferability, without extra computational cost.

Findings

SAAD improves AutoAttack robustness on CIFAR-10, CIFAR-100, and Tiny-ImageNet.

Stronger teachers do not always produce more robust students, highlighting the robustness saturation phenomenon.

Transferability of adversarial examples is crucial for effective robustness transfer.

Abstract

Adversarial distillation in the standard min-max adversarial training framework aims to transfer adversarial robustness from a large, robust teacher network to a compact student. However, existing work often neglects to incorporate state-of-the-art robust teachers. Through extensive analysis, we find that stronger teachers do not necessarily yield more robust students-a phenomenon known as robust saturation. While typically attributed to capacity gaps, we show that such explanations are incomplete. Instead, we identify adversarial transferability-the fraction of student-crafted adversarial examples that remain effective against the teacher-as a key factor in successful robustness transfer. Based on this insight, we propose Sample-wise Adaptive Adversarial Distillation (SAAD), which reweights training examples by their measured transferability without incurring additional computational cost. Experiments on CIFAR-10, CIFAR-100, and Tiny-ImageNet show that SAAD consistently improves AutoAttack robustness over prior methods. Our code is available at https://github.com/HongsinLee/saad.