FastAT Benchmark: A Comprehensive Framework for Fair Evaluation of Fast Adversarial Training Methods

TL;DR

The paper introduces a standardized benchmark for evaluating Fast Adversarial Training methods, enabling fair comparison of robustness and efficiency across diverse techniques.

Contribution

It presents a controlled, unified evaluation framework with over twenty methods, standard settings, and comprehensive metrics for robustness and computational cost.

Findings

Single-step methods can match or outperform PGD-AT in robustness at lower cost.

No single FastAT method dominates across all evaluation metrics.

The benchmark facilitates transparent, reproducible comparison of FastAT techniques.

Abstract

Fast Adversarial Training (FastAT) seeks to achieve adversarial robustness at a fraction of the computational cost incurred by standard multi-step methods such as PGD-AT. Although numerous FastAT techniques have been proposed in recent years, fair comparison among them remains elusive. Existing benchmarks and public leaderboards typically permit diverse model architectures, varying training configurations, and external data sources, making it unclear whether reported improvements reflect genuine algorithmic advances or merely more favorable experimental conditions. To address this problem, we introduce the FastAT Benchmark, a controlled evaluation framework built on three core design principles: unified architecture requirements, standardized training settings, and strict prohibition of external or synthetic data. The benchmark implements over twenty representative FastAT methods within a single codebase, enabling direct and reproducible comparison. Each method is assessed through a dual-metric evaluation framework that measures both adversarial robustness (accuracy under PGD, AutoAttack, and CR Attack) and computational cost (GPU training time and peak memory footprint). Comprehensive experiments on CIFAR-10, CIFAR-100, and Tiny-ImageNet provide reliable baseline measurements and reveal that well-designed single-step methods can match or surpass PGD-AT robustness at substantially lower cost, while no single method dominates across all evaluation dimensions. The complete benchmark, including source code, configuration files, and experimental results, is publicly available to support transparent and fair evaluation of future FastAT research.