PADetBench: Towards Benchmarking Physical Attacks against Object Detection
Jiawei Lian, Jianhong Pan, Lefan Wang, Yi Wang, Lap-Pui Chau, Shaohui Mei
TL;DR
This paper introduces PADetBench, a comprehensive simulation-based benchmark for evaluating physical attacks on object detection models, enabling fair, large-scale, and controlled experiments that address real-world challenges.
Contribution
It presents a novel benchmark framework using realistic simulation to evaluate physical attacks on object detection, including extensive experiments and analysis.
Findings
Physical attack performance varies significantly across methods.
Simulation enables large-scale, controlled evaluation of physical attacks.
Insights into physical adversarial robustness and future research directions.
Abstract
Physical attacks against object detection have gained increasing attention due to their significant practical implications. However, conducting physical experiments is extremely time-consuming and labor-intensive. Moreover, physical dynamics and cross-domain transformation are challenging to strictly regulate in the real world, leading to unaligned evaluation and comparison, severely hindering the development of physically robust models. To accommodate these challenges, we explore utilizing realistic simulation to thoroughly and rigorously benchmark physical attacks with fairness under controlled physical dynamics and cross-domain transformation. This resolves the problem of capturing identical adversarial images that cannot be achieved in the real world. Our benchmark includes 20 physical attack methods, 48 object detectors, comprehensive physical dynamics, and evaluation metrics. We…
Peer Reviews
Decision·ICLR 2025 Conference Withdrawn Submission
1. The goal of this paper is significant to this area. 2. The authors evaluated attacks under multiple metrics and discussed the strength of some metrics. 3. The authors conducted comprehensive evaluations.
1. Some figures are hard to read, e.g. Fig 1. Some of the text is too small and blurred. 2. Some settings seem problematic. Please see the questions. 3. The paper lacks conclusions and inspirations drawn from benchmarking multiple attacks. For example, which technique is essential in improving adversarial effectiveness?
- Moving toward better evaluations of physical attacks in real-world conditions is important and interesting. - A large set of physical attacks and object detectors are evaluated.
Despite creating a standard benchmark to compare what may be the performance of physical attacks in real-world scenarios, this paper falls short of creating such a benchmark. Too many details are missing, and the presentation could be significantly improved. Please find bellow the different weaknesses that I find necessary to address: - A lack of comparison with other benchmarks of the literature. For example, Zhang et al. (2023b) generated the DCI dataset using the CARLA simulator. What is the
This work provides extensive experiments to support the proposed idea.
I appreciate that the authors present extensive experimental results to assess the performance of physical adversarial attacks and address the shortcomings of current benchmarks, including their time-consuming and costly nature, challenges in aligning physical dynamics, cross-domain loss, and difficulties in comparison (lines 46-53). However, readers unfamiliar with physical adversarial attacks may struggle to understand the significance of these issues and how the proposed benchmark effectively
1. The paper shows great efforts to organize such a large benchmark and provide detailed analysis. 2. Overall, the paper is well-written and almost clear to me. 3. Analysis tools in the benchmark are helpful to many researchers, and the user feedbacks reflect its ease of use. 4. The discussion about "where are we" and "where to go" is interesting and inspiring.
1. The presentation can be improved. Although comprehensive experimental results are necessary, some crowd the layout and cause a terrible experience. For instance, Figure 2 provides extensive results while each sub-figure holds only a small subset of space. It makes readers try their best to broaden the figure and keep their eyes fixed on it. Selectively displaying results may be a better choice. 2. The comparison of other benchmarks in object detection should be available. Since the benchmark
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdversarial Robustness in Machine Learning · Advanced Malware Detection Techniques · Anomaly Detection Techniques and Applications
MethodsSoftmax · Attention Is All You Need