Benchmarking Adversarial Robustness

TL;DR

This paper presents a comprehensive benchmark for evaluating adversarial robustness in image classification, systematically comparing attack and defense methods to guide future research.

Contribution

It introduces a rigorous benchmark with large-scale experiments and robustness curves for fair evaluation of adversarial attack and defense techniques.

Findings

Identified strengths and weaknesses of current methods

Provided insights for improving adversarial robustness

Established a standard evaluation framework

Abstract

Deep neural networks are vulnerable to adversarial examples, which becomes one of the most important research problems in the development of deep learning. While a lot of efforts have been made in recent years, it is of great significance to perform correct and complete evaluations of the adversarial attack and defense algorithms. In this paper, we establish a comprehensive, rigorous, and coherent benchmark to evaluate adversarial robustness on image classification tasks. After briefly reviewing plenty of representative attack and defense methods, we perform large-scale experiments with two robustness curves as the fair-minded evaluation criteria to fully understand the performance of these methods. Based on the evaluation results, we draw several important findings and provide insights for future research.