A Black-Box Evaluation Framework for Semantic Robustness in Bird's Eye View Detection

TL;DR

This paper introduces a black-box evaluation framework for assessing the semantic robustness of Bird's Eye View detection models against adversarial semantic perturbations, highlighting vulnerabilities and robustness variations among models.

Contribution

It presents the first adversarial optimization approach for semantic perturbations in BEV detection and provides a benchmark comparing ten recent models' robustness.

Findings

PolarFormer shows highest robustness among tested models.

BEVDet's performance drops to zero under certain perturbations.

The framework effectively identifies model vulnerabilities.

Abstract

Camera-based Bird's Eye View (BEV) perception models receive increasing attention for their crucial role in autonomous driving, a domain where concerns about the robustness and reliability of deep learning have been raised. While only a few works have investigated the effects of randomly generated semantic perturbations, aka natural corruptions, on the multi-view BEV detection task, we develop a black-box robustness evaluation framework that adversarially optimises three common semantic perturbations: geometric transformation, colour shifting, and motion blur, to deceive BEV models, serving as the first approach in this emerging field. To address the challenge posed by optimising the semantic perturbation, we design a smoothed, distance-based surrogate function to replace the mAP metric and introduce SimpleDIRECT, a deterministic optimisation algorithm that utilises observed slopes to guide the optimisation process. By comparing with randomised perturbation and two optimisation baselines, we demonstrate the effectiveness of the proposed framework. Additionally, we provide a benchmark on the semantic robustness of ten recent BEV models. The results reveal that PolarFormer, which emphasises geometric information from multi-view images, exhibits the highest robustness, whereas BEVDet is fully compromised, with its precision reduced to zero.