Towards Universal Physical Adversarial Attacks via a Joint Multi-Objective and Multi-Model Optimization Framework

TL;DR

This paper introduces JMOF, a novel framework for physical adversarial attacks that enhances transferability and generalization across multiple models and vision tasks by addressing gradient conflicts and model ensemble selection.

Contribution

The paper proposes a joint multi-objective, multi-model optimization framework with a dual-level mechanism and orthogonal gradient alignment to improve physical attack transferability and cross-task generalization.

Findings

JMOF outperforms state-of-the-art baselines in simulated and real-world experiments.

JMOF generates attacks that deceive multiple vision models simultaneously.

The framework demonstrates strong cross-vision-task generalization, affecting detection, segmentation, and depth estimation.

Abstract

Physical adversarial attacks often overfit single surrogate models and optimization objectives. While ensemble attacks can mitigate this, existing methods struggle with severe gradient conflicts within restricted physical texture spaces, significantly degrading cross-model transferability. To bridge this gap, this paper proposes a Joint Multi-Objective and Multi-Model Optimization Framework (JMOF) that leverages quantitative similarity analysis to select the optimal surrogate model ensemble. Within JMOF, a dual-level mechanism jointly suppresses prediction outputs and flattens intermediate feature distributions, balancing attack efficiency with deep generalization. Additionally, an Orthogonal Gradient Alignment (OGA) strategy resolves cross-model gradient conflicts, transforming mutually repulsive gradients into synergistic optimization directions. Extensive simulated and real-world experiments demonstrate that JMOF outperforms state-of-the-art baselines against diverse black-box detectors. Crucially, JMOF exhibits substantial cross-vision-task generalization, generating attacks capable of simultaneously deceiving object detection and semantic segmentation or monocular depth estimation models. This research advances the generalization limits of physical adversarial attacks, providing a robust framework for evaluating visual AI vulnerabilities in real-world deployments.