In-distribution adversarial attacks on object recognition models using gradient-free search

TL;DR

This paper demonstrates that neural networks can be fooled by in-distribution adversarial examples created through gradient-free search, revealing a significant vulnerability even within the training data distribution.

Contribution

The authors introduce CMA-Search, a gradient-free evolution strategy method to find in-distribution adversarial examples, challenging the assumption that such errors only occur outside the training distribution.

Findings

CMA-Search finds in-distribution adversarial examples in over 71% of camera perturbation cases.

Lighting perturbations cause misclassifications in 42% of cases.

The phenomenon extends to natural images from ImageNet and Co3D datasets.

Abstract

Neural networks are susceptible to small perturbations in the form of 2D rotations and shifts, image crops, and even changes in object colors. Past works attribute these errors to dataset bias, claiming that models fail on these perturbed samples as they do not belong to the training data distribution. Here, we challenge this claim and present evidence of the widespread existence of perturbed images within the training data distribution, which networks fail to classify. We train models on data sampled from parametric distributions, then search inside this data distribution to find such in-distribution adversarial examples. This is done using our gradient-free evolution strategies (ES) based approach which we call CMA-Search. Despite training with a large-scale (0.5 million images), unbiased dataset of camera and light variations, CMA-Search can find a failure inside the data distribution in over 71% cases by perturbing the camera position. With lighting changes, CMA-Search finds misclassifications in 42% cases. These findings also extend to natural images from ImageNet and Co3D datasets. This phenomenon of in-distribution images presents a highly worrisome problem for artificial intelligence -- they bypass the need for a malicious agent to add engineered noise to induce an adversarial attack. All code, datasets, and demos are available at https://github.com/Spandan-Madan/in_distribution_adversarial_examples.