Benchmarking Adversarial Patch Selection and Location

TL;DR

This paper introduces PatchMap, a comprehensive benchmark for adversarial patch placement on images, revealing vulnerable regions and improving attack success rates through a segmentation-guided heuristic, aiding defense and attack research.

Contribution

The paper presents PatchMap, the first exhaustive spatial benchmark for patch placement, and a segmentation-based heuristic to identify vulnerable regions without gradient queries.

Findings

PatchMap evaluated over 1.5e8 forward passes on ImageNet.

Small patches as little as 2% of the image can cause misclassification.

The heuristic improves attack success rates by 8-13 percentage points.

Abstract

Adversarial patch attacks threaten the reliability of modern vision models. We present PatchMap, the first spatially exhaustive benchmark of patch placement, built by evaluating over 1.5e8 forward passes on ImageNet validation images. PatchMap reveals systematic hot-spots where small patches (as little as 2% of the image) induce confident misclassifications and large drops in model confidence. To demonstrate its utility, we propose a simple segmentation guided placement heuristic that leverages off the shelf masks to identify vulnerable regions without any gradient queries. Across five architectures-including adversarially trained ResNet50, our method boosts attack success rates by 8 to 13 percentage points compared to random or fixed placements. We publicly release PatchMap and the code implementation. The full PatchMap bench (6.5B predictions, multiple backbones) will be released soon to further accelerate research on location-aware defenses and adaptive attacks.