A Comprehensive Study on Robustness of Image Classification Models: Benchmarking and Rethinking

TL;DR

This paper introduces ARES-Bench, a comprehensive robustness benchmark for image classification models, evaluating 55 models across various attacks and datasets, revealing key trade-offs and improvements in robustness techniques.

Contribution

The paper establishes a large-scale robustness benchmark for image classification, providing extensive evaluations and insights into model robustness, and achieves state-of-the-art adversarial robustness through optimized training settings.

Findings

Inherent trade-off between adversarial and natural robustness.

Adversarial training enhances robustness, especially for Transformers.

Pre-training improves natural robustness with more data or self-supervised learning.

Abstract

The robustness of deep neural networks is usually lacking under adversarial examples, common corruptions, and distribution shifts, which becomes an important research problem in the development of deep learning. Although new deep learning methods and robustness improvement techniques have been constantly proposed, the robustness evaluations of existing methods are often inadequate due to their rapid development, diverse noise patterns, and simple evaluation metrics. Without thorough robustness evaluations, it is hard to understand the advances in the field and identify the effective methods. In this paper, we establish a comprehensive robustness benchmark called \textbf{ARES-Bench} on the image classification task. In our benchmark, we evaluate the robustness of 55 typical deep learning models on ImageNet with diverse architectures (e.g., CNNs, Transformers) and learning algorithms (e.g., normal supervised training, pre-training, adversarial training) under numerous adversarial attacks and out-of-distribution (OOD) datasets. Using robustness curves as the major evaluation criteria, we conduct large-scale experiments and draw several important findings, including: 1) there is an inherent trade-off between adversarial and natural robustness for the same model architecture; 2) adversarial training effectively improves adversarial robustness, especially when performed on Transformer architectures; 3) pre-training significantly improves natural robustness based on more training data or self-supervised learning. Based on ARES-Bench, we further analyze the training tricks in large-scale adversarial training on ImageNet. By designing the training settings accordingly, we achieve the new state-of-the-art adversarial robustness. We have made the benchmarking results and code platform publicly available.