RobustART: Benchmarking Robustness on Architecture Design and Training Techniques

TL;DR

RobustART is a comprehensive benchmark evaluating how different neural network architectures and training techniques influence robustness against various noises, providing new insights and an open-source platform for developing more resilient DNNs.

Contribution

It introduces RobustART, the first extensive benchmark on ImageNet assessing architecture and training impacts on robustness, with new insights into model performance and an open-source evaluation platform.

Findings

Adversarial training improves robustness across noise types for Transformers and MLP-Mixers.

CNNs outperform Transformers and MLP-Mixers on natural and system noises; Transformers excel against adversarial noise.

Increasing model size or data does not always enhance robustness for lightweight architectures.

Abstract

Deep neural networks (DNNs) are vulnerable to adversarial noises, which motivates the benchmark of model robustness. Existing benchmarks mainly focus on evaluating defenses, but there are no comprehensive studies of how architecture design and training techniques affect robustness. Comprehensively benchmarking their relationships is beneficial for better understanding and developing robust DNNs. Thus, we propose RobustART, the first comprehensive Robustness investigation benchmark on ImageNet regarding ARchitecture design (49 human-designed off-the-shelf architectures and 1200+ networks from neural architecture search) and Training techniques (10+ techniques, e.g., data augmentation) towards diverse noises (adversarial, natural, and system noises). Extensive experiments substantiated several insights for the first time, e.g., (1) adversarial training is effective for the robustness against all noises types for Transformers and MLP-Mixers; (2) given comparable model sizes and aligned training settings, CNNs > Transformers > MLP-Mixers on robustness against natural and system noises; Transformers > MLP-Mixers > CNNs on adversarial robustness; (3) for some light-weight architectures, increasing model sizes or using extra data cannot improve robustness. Our benchmark presents: (1) an open-source platform for comprehensive robustness evaluation; (2) a variety of pre-trained models to facilitate robustness evaluation; and (3) a new view to better understand the mechanism towards designing robust DNNs. We will continuously develop to this ecosystem for the community.