A Data-Centric Approach for Improving Adversarial Training Through the Lens of Out-of-Distribution Detection

TL;DR

This paper introduces a data-centric approach to enhance adversarial training by detecting and removing hard samples using out-of-distribution detection techniques, leading to improved robustness with minimal computational overhead.

Contribution

It proposes a novel method that identifies and removes hard training samples based on out-of-distribution detection, simplifying adversarial training improvements.

Findings

Effective detection of hard samples using maximum softmax probability

Improved adversarial robustness on SVHN and CIFAR-10 datasets

Reduced computational cost compared to existing methods

Abstract

Current machine learning models achieve super-human performance in many real-world applications. Still, they are susceptible against imperceptible adversarial perturbations. The most effective solution for this problem is adversarial training that trains the model with adversarially perturbed samples instead of original ones. Various methods have been developed over recent years to improve adversarial training such as data augmentation or modifying training attacks. In this work, we examine the same problem from a new data-centric perspective. For this purpose, we first demonstrate that the existing model-based methods can be equivalent to applying smaller perturbation or optimization weights to the hard training examples. By using this finding, we propose detecting and removing these hard samples directly from the training procedure rather than applying complicated algorithms to mitigate their effects. For detection, we use maximum softmax probability as an effective method in out-of-distribution detection since we can consider the hard samples as the out-of-distribution samples for the whole data distribution. Our results on SVHN and CIFAR-10 datasets show the effectiveness of this method in improving the adversarial training without adding too much computational cost.