T2I-Based Physical-World Appearance Attack against Traffic Sign Recognition Systems in Autonomous Driving

TL;DR

This paper introduces DiffSign, a novel T2I-based physical-world attack framework that creates stealthy, transferable adversarial traffic sign appearances to deceive traffic sign recognition systems in autonomous driving, with high success rates.

Contribution

The paper presents DiffSign, a new T2I diffusion model-based attack method with CLIP-guided loss and style customization, improving stealthiness, transferability, and robustness over prior adversarial approaches.

Findings

Achieves 83.3% attack success rate in real-world conditions.

Demonstrates high transferability across different TSR models.

Enhances attack stealthiness and generalization through style customization.

Abstract

Traffic Sign Recognition (TSR) systems play a critical role in Autonomous Driving (AD) systems, enabling real-time detection of road signs, such as STOP and speed limit signs. While these systems are increasingly integrated into commercial vehicles, recent research has exposed their vulnerability to physical-world adversarial appearance attacks. In such attacks, carefully crafted visual patterns are misinterpreted by TSR models as legitimate traffic signs, while remaining inconspicuous or benign to human observers. However, existing adversarial appearance attacks suffer from notable limitations. Pixel-level perturbation-based methods often lack stealthiness and tend to overfit to specific surrogate models, resulting in poor transferability to real-world TSR systems. On the other hand, text-to-image (T2I) diffusion model-based approaches demonstrate limited effectiveness and poor generalization to out-of-distribution sign types. In this paper, we present DiffSign, a novel T2I-based appearance attack framework designed to generate physically robust, highly effective, transferable, practical, and stealthy appearance attacks against TSR systems. To overcome the limitations of prior approaches, we propose a carefully designed attack pipeline that integrates CLIP-based loss and masked prompts to improve attack focus and controllability. We also propose two novel style customization methods to guide visual appearance and improve out-of-domain traffic sign attack generalization and attack stealthiness. We conduct extensive evaluations of DiffSign under varied real-world conditions, including different distances, angles, light conditions, and sign categories. Our method achieves an average physical-world attack success rate of 83.3%, leveraging DiffSign's high effectiveness in attack transferability.