Revisiting Physical-World Adversarial Attack on Traffic Sign Recognition: A Commercial Systems Perspective

TL;DR

This study evaluates the effectiveness of physical-world adversarial attacks on commercial Traffic Sign Recognition systems, revealing their variable success rates and the influence of system design features like spatial memorization.

Contribution

It provides the first large-scale measurement of such attacks on real-world commercial TSR systems and introduces new metrics to better understand attack success factors.

Findings

High attack success (100%) on some commercial systems

Lower overall success rates due to system design factors

Seven novel insights challenging prior assumptions

Abstract

Traffic Sign Recognition (TSR) is crucial for safe and correct driving automation. Recent works revealed a general vulnerability of TSR models to physical-world adversarial attacks, which can be low-cost, highly deployable, and capable of causing severe attack effects such as hiding a critical traffic sign or spoofing a fake one. However, so far existing works generally only considered evaluating the attack effects on academic TSR models, leaving the impacts of such attacks on real-world commercial TSR systems largely unclear. In this paper, we conduct the first large-scale measurement of physical-world adversarial attacks against commercial TSR systems. Our testing results reveal that it is possible for existing attack works from academia to have highly reliable (100\%) attack success against certain commercial TSR system functionality, but such attack capabilities are not generalizable, leading to much lower-than-expected attack success rates overall. We find that one potential major factor is a spatial memorization design that commonly exists in today's commercial TSR systems. We design new attack success metrics that can mathematically model the impacts of such design on the TSR system-level attack success, and use them to revisit existing attacks. Through these efforts, we uncover 7 novel observations, some of which directly challenge the observations or claims in prior works due to the introduction of the new metrics.