Auto-ART: Structured Literature Synthesis and Automated Adversarial Robustness Testing

TL;DR

Auto-ART is a comprehensive framework that synthesizes literature on adversarial robustness and provides automated, multi-norm testing and detection of gradient masking to improve trustworthy ML deployment.

Contribution

It offers the first structured literature analysis combined with an open-source framework supporting extensive adversarial robustness evaluation.

Findings

Auto-ART's pre-screening detects gradient masking in 92% of cases.

RDI rankings strongly correlate with full AutoAttack results.

Multi-norm evaluation reveals a 23.5 percentage point robustness gap.

Abstract

Adversarial robustness evaluation underpins every claim of trustworthy ML deployment, yet the field suffers from fragmented protocols and undetected gradient masking. We make two contributions. (1) Structured synthesis. We analyze nine peer-reviewed corpus sources (2020--2026) through seven complementary protocols, producing the first end-to-end structured analysis of the field's consensus and unresolved challenges. (2) Auto-ART framework. We introduce Auto-ART, an open-source framework that operationalizes identified gaps: 50+ attacks, 28 defense modules, the Robustness Diagnostic Index (RDI), and gradient-masking detection. It supports multi-norm evaluation (l1/l2/linf/semantic/spatial) and compliance mapping to NIST AI RMF, OWASP LLM Top 10, and the EU AI Act. Empirical validation on RobustBench demonstrates that Auto-ART's pre-screening identifies gradient masking in 92% of flagged cases, and RDI rankings correlate highly with full AutoAttack. Multi-norm evaluation exposes a 23.5 pp gap between average and worst-case robustness on state-of-the-art models. No prior work combines such structured meta-scientific analysis with an executable evaluation framework bridging literature gaps into engineering.