Dual Head Adversarial Training

TL;DR

This paper introduces Dual Head Adversarial Training (DH-AT), a novel method that enhances the robustness of adversarially trained neural networks by modifying architecture and training strategy, leading to improved defense against attacks.

Contribution

DH-AT is the first approach to attach a second network head and use a CNN aggregator, significantly improving robustness over existing adversarial training methods.

Findings

DH-AT improves robustness by 3.4% against PGD40.

DH-AT enhances AutoAttack robustness by 2.3%.

DH-AT increases clean accuracy by 1.8%.

Abstract

Deep neural networks (DNNs) are known to be vulnerable to adversarial examples/attacks, raising concerns about their reliability in safety-critical applications. A number of defense methods have been proposed to train robust DNNs resistant to adversarial attacks, among which adversarial training has so far demonstrated the most promising results. However, recent studies have shown that there exists an inherent tradeoff between accuracy and robustness in adversarially-trained DNNs. In this paper, we propose a novel technique Dual Head Adversarial Training (DH-AT) to further improve the robustness of existing adversarial training methods. Different from existing improved variants of adversarial training, DH-AT modifies both the architecture of the network and the training strategy to seek more robustness. Specifically, DH-AT first attaches a second network head (or branch) to one intermediate layer of the network, then uses a lightweight convolutional neural network (CNN) to aggregate the outputs of the two heads. The training strategy is also adapted to reflect the relative importance of the two heads. We empirically show, on multiple benchmark datasets, that DH-AT can bring notable robustness improvements to existing adversarial training methods. Compared with TRADES, one state-of-the-art adversarial training method, our DH-AT can improve the robustness by 3.4% against PGD40 and 2.3% against AutoAttack, and also improve the clean accuracy by 1.8%.