Benchmarking Robustness of Deep Learning Classifiers Using Two-Factor Perturbation

TL;DR

This paper benchmarks the robustness of deep learning classifiers against complex two-factor perturbations, introducing new evaluation tools and demonstrating improved robustness with two-factor perturbations over single-factor ones.

Contribution

It introduces a comprehensive two-factor perturbation benchmarking framework, including new evaluation metrics and visualization tools, to better assess DL classifier robustness.

Findings

Two-factor perturbations improve classifier robustness and accuracy.

New statistical matrix and visualization tools enhance robustness evaluation.

Benchmarking datasets and code are publicly available for research.

Abstract

Accuracies of deep learning (DL) classifiers are often unstable in that they may change significantly when retested on adversarial images, imperfect images, or perturbed images. This paper adds to the fundamental body of work on benchmarking the robustness of DL classifiers on defective images. To measure robust DL classifiers, previous research reported on single-factor corruption. We created comprehensive 69 benchmarking image sets, including a clean set, sets with single factor perturbations, and sets with two-factor perturbation conditions. The state-of-the-art two-factor perturbation includes (a) two digital perturbations (salt & pepper noise and Gaussian noise) applied in both sequences, and (b) one digital perturbation (salt & pepper noise) and a geometric perturbation (rotation) applied in both sequences. Previous research evaluating DL classifiers has often used top-1/top-5 accuracy. We innovate a new two-dimensional, statistical matrix to evaluating robustness of DL classifiers. Also, we introduce a new visualization tool, including minimum accuracy, maximum accuracy, mean accuracies, and coefficient of variation (CV), for benchmarking robustness of DL classifiers. Comparing with single factor corruption, we first report that using two-factor perturbed images improves both robustness and accuracy of DL classifiers. All source codes and related image sets are shared on the Website at http://cslinux.semo.edu/david/data to support future academic research and industry projects.