Robust Assessment of Real-World Adversarial Examples

TL;DR

This paper introduces a systematic framework for evaluating real-world adversarial examples, highlighting the importance of considering environmental factors and proposing a new scoring method for more comprehensive assessment.

Contribution

It presents a testbed, a new evaluation score, and additional assessments to improve the robustness testing of adversarial examples in real-world scenarios.

Findings

Significant performance differences under small environmental perturbations.

Current reporting methods lack consideration of scene changes.

Proposed score offers a more complete evaluation of adversarial robustness.

Abstract

We explore rigorous, systematic, and controlled experimental evaluation of adversarial examples in the real world and propose a testing regimen for evaluation of real world adversarial objects. We show that for small scene/ environmental perturbations, large adversarial performance differences exist. Current state of adversarial reporting exists largely as a frequency count over a dynamic collections of scenes. Our work underscores the need for either a more complete report or a score that incorporates scene changes and baseline performance for models and environments tested by adversarial developers. We put forth a score that attempts to address the above issues in a straight-forward exemplar application for multiple generated adversary examples. We contribute the following: 1. a testbed for adversarial assessment, 2. a score for adversarial examples, and 3. a collection of additional evaluations on testbed data.