Batch-in-Batch: a new adversarial training framework for initial perturbation and sample selection

TL;DR

This paper introduces Batch-in-Batch, a novel adversarial training framework that enhances model robustness by diversifying initial perturbations and employing sample selection strategies, leading to significant accuracy improvements.

Contribution

The paper proposes a new training framework that jointly constructs multiple perturbation sets and uses sample selection to improve adversarial robustness, validated on multiple datasets and models.

Findings

Over 13% accuracy improvement on SVHN with attack radius 8/255.

Consistent higher adversarial accuracy across datasets and models.

Cost-effective framework with large m value.

Abstract

Adversarial training methods commonly generate independent initial perturbation for adversarial samples from a simple uniform distribution, and obtain the training batch for the classifier without selection. In this work, we propose a simple yet effective training framework called Batch-in-Batch (BB) to enhance models robustness. It involves specifically a joint construction of initial values that could simultaneously generates $m$ sets of perturbations from the original batch set to provide more diversity for adversarial samples; and also includes various sample selection strategies that enable the trained models to have smoother losses and avoid overconfident outputs. Through extensive experiments on three benchmark datasets (CIFAR-10, SVHN, CIFAR-100) with two networks (PreActResNet18 and WideResNet28-10) that are used in both the single-step (Noise-Fast Gradient Sign Method, N-FGSM) and multi-step (Projected Gradient Descent, PGD-10) adversarial training, we show that models trained within the BB framework consistently have higher adversarial accuracy across various adversarial settings, notably achieving over a 13% improvement on the SVHN dataset with an attack radius of 8/255 compared to the N-FGSM baseline model. Furthermore, experimental analysis of the efficiency of both the proposed initial perturbation method and sample selection strategies validates our insights. Finally, we show that our framework is cost-effective in terms of computational resources, even with a relatively large value of $m$.