RoBIC: A benchmark suite for assessing classifiers robustness

TL;DR

RoBIC is a new benchmark suite that objectively and efficiently evaluates the robustness of image classifiers against adversarial attacks using a novel half-distortion measure.

Contribution

It introduces RoBIC, a parameter-free, faster benchmark that assesses classifier robustness independently of accuracy and compares 16 recent models.

Findings

RoBIC effectively differentiates model robustness.

RoBIC is faster than existing benchmarks.

Significant robustness differences found among recent models.

Abstract

Many defenses have emerged with the development of adversarial attacks. Models must be objectively evaluated accordingly. This paper systematically tackles this concern by proposing a new parameter-free benchmark we coin RoBIC. RoBIC fairly evaluates the robustness of image classifiers using a new half-distortion measure. It gauges the robustness of the network against white and black box attacks, independently of its accuracy. RoBIC is faster than the other available benchmarks. We present the significant differences in the robustness of 16 recent models as assessed by RoBIC.