Benchmarking Robustness of Deep Learning Classifiers Using Two-Factor Perturbation

TL;DR

This paper introduces a new benchmarking methodology and visualization tool for evaluating the robustness of deep learning classifiers against two-factor perturbations, demonstrating improved robustness and accuracy.

Contribution

It presents a novel two-factor perturbation benchmarking approach and a four-quadrant visualization tool for assessing deep learning classifier robustness.

Findings

Two-factor perturbations improve classifier robustness and accuracy.

The new benchmarking method effectively differentiates classifier performance.

Shared resources support future research and development.

Abstract

This paper adds to the fundamental body of work on benchmarking the robustness of deep learning (DL) classifiers. We innovate a new benchmarking methodology to evaluate robustness of DL classifiers. Also, we introduce a new four-quadrant statistical visualization tool, including minimum accuracy, maximum accuracy, mean accuracy, and coefficient of variation, for benchmarking robustness of DL classifiers. To measure robust DL classifiers, we created a comprehensive 69 benchmarking image set, including a clean set, sets with single factor perturbations, and sets with two-factor perturbation conditions. After collecting experimental results, we first report that using two-factor perturbed images improves both robustness and accuracy of DL classifiers. The two-factor perturbation includes (1) two digital perturbations (salt & pepper noise and Gaussian noise) applied in both sequences, and (2) one digital perturbation (salt & pepper noise) and a geometric perturbation (rotation) applied in both sequences. All source codes, related image sets, and preliminary data, figures are shared on a GitHub website to support future academic research and industry projects. The web resources locate at https://github.com/caperock/robustai