Towards Class-wise Fair Adversarial Training via Anti-Bias Soft Label Distillation

TL;DR

This paper introduces Anti-Bias Soft Label Distillation (ABSLD), a novel method that improves adversarial robustness fairness across classes by adaptively adjusting soft label smoothness during training, outperforming existing methods.

Contribution

The paper proposes ABSLD, a new knowledge distillation approach that enhances adversarial robustness fairness by class-wise soft label adjustment, supported by empirical and theoretical analysis.

Findings

ABSLD improves robustness and fairness simultaneously.

Class-wise soft label smoothness significantly impacts robustness fairness.

ABSLD outperforms state-of-the-art methods in experiments.

Abstract

Adversarial Training (AT) is widely recognized as an effective approach to enhance the adversarial robustness of Deep Neural Networks. As a variant of AT, Adversarial Robustness Distillation (ARD) has shown outstanding performance in enhancing the robustness of small models. However, both AT and ARD face robust fairness issue: these models tend to display strong adversarial robustness against some classes (easy classes) while demonstrating weak adversarial robustness against others (hard classes). This paper explores the underlying factors of this problem and points out the smoothness degree of soft labels for different classes significantly impacts the robust fairness from both empirical observation and theoretical analysis. Based on the above exploration, we propose Anti-Bias Soft Label Distillation (ABSLD) within the Knowledge Distillation framework to enhance the adversarial robust fairness. Specifically, ABSLD adaptively reduces the student's error risk gap between different classes, which is accomplished by adjusting the class-wise smoothness degree of teacher's soft labels during the training process, and the adjustment is managed by assigning varying temperatures to different classes. Additionally, as a label-based approach, ABSLD is highly adaptable and can be integrated with the sample-based methods. Extensive experiments demonstrate ABSLD outperforms state-of-the-art methods on the comprehensive performance of robustness and fairness.