Adversarial Attention Perturbations for Large Object Detection Transformers

TL;DR

This paper introduces AFOG, a neural-architecture agnostic adversarial attack that effectively exposes vulnerabilities in large object detection transformers and CNN-based detectors by focusing perturbations on critical image regions.

Contribution

AFOG employs a learnable attention mechanism and integrated feature loss to generate stealthy adversarial perturbations, significantly improving attack success over existing methods.

Findings

AFOG increases attack performance by up to 30.6% over non-attention baselines.

AFOG outperforms existing attacks by up to 83% on large detection transformers.

AFOG is efficient, stealthy, and effective against both transformer-based and CNN-based detectors.

Abstract

Adversarial perturbations are useful tools for exposing vulnerabilities in neural networks. Existing adversarial perturbation methods for object detection are either limited to attacking CNN-based detectors or weak against transformer-based detectors. This paper presents an Attention-Focused Offensive Gradient (AFOG) attack against object detection transformers. By design, AFOG is neural-architecture agnostic and effective for attacking both large transformer-based object detectors and conventional CNN-based detectors with a unified adversarial attention framework. This paper makes three original contributions. First, AFOG utilizes a learnable attention mechanism that focuses perturbations on vulnerable image regions in multi-box detection tasks, increasing performance over non-attention baselines by up to 30.6%. Second, AFOG's attack loss is formulated by integrating two types of feature loss through learnable attention updates with iterative injection of adversarial perturbations. Finally, AFOG is an efficient and stealthy adversarial perturbation method. It probes the weak spots of detection transformers by adding strategically generated and visually imperceptible perturbations which can cause well-trained object detection models to fail. Extensive experiments conducted with twelve large detection transformers on COCO demonstrate the efficacy of AFOG. Our empirical results also show that AFOG outperforms existing attacks on transformer-based and CNN-based object detectors by up to 83% with superior speed and imperceptibility. Code is available at https://github.com/zacharyyahn/AFOG.