Benchmarking Neural Network Robustness to Common Corruptions and Perturbations

TL;DR

This paper introduces standardized benchmarks, ImageNet-C and ImageNet-P, to evaluate image classifier robustness against common corruptions and perturbations, aiding development of more reliable neural networks.

Contribution

It presents new benchmarks for corruption and perturbation robustness, expanding evaluation beyond adversarial attacks, and explores methods to improve classifier robustness.

Findings

Negligible robustness difference between AlexNet and ResNet.

Certain adversarial defenses also improve robustness to common perturbations.

Benchmarks facilitate future research on robust neural network generalization.

Abstract

In this paper we establish rigorous benchmarks for image classifier robustness. Our first benchmark, ImageNet-C, standardizes and expands the corruption robustness topic, while showing which classifiers are preferable in safety-critical applications. Then we propose a new dataset called ImageNet-P which enables researchers to benchmark a classifier's robustness to common perturbations. Unlike recent robustness research, this benchmark evaluates performance on common corruptions and perturbations not worst-case adversarial perturbations. We find that there are negligible changes in relative corruption robustness from AlexNet classifiers to ResNet classifiers. Afterward we discover ways to enhance corruption and perturbation robustness. We even find that a bypassed adversarial defense provides substantial common perturbation robustness. Together our benchmarks may aid future work toward networks that robustly generalize.