Crafting Physical Adversarial Examples by Combining Differentiable and Physically Based Renders

TL;DR

This paper introduces PAV-Camou, a novel method combining differentiable and physically based rendering techniques to generate photorealistic adversarial camouflage for vehicles, enhancing robustness in real-world autonomous driving scenarios.

Contribution

The paper presents a new approach that adjusts coordinate mapping and combines two renderers to produce effective, physically realizable adversarial camouflage for vehicles.

Findings

Effective in digital and physical environments

Improves robustness of adversarial camouflage

Achieves photorealism under diverse conditions

Abstract

Recently we have witnessed progress in hiding road vehicles against object detectors through adversarial camouflage in the digital world. The extension of this technique to the physical world is crucial for testing the robustness of autonomous driving systems. However, existing methods do not show good performances when applied to the physical world. This is partly due to insufficient photorealism in training examples, and lack of proper physical realization methods for camouflage. To generate a robust adversarial camouflage suitable for real vehicles, we propose a novel method called PAV-Camou. We propose to adjust the mapping from the coordinates in the 2D map to those of corresponding 3D model. This process is critical for mitigating texture distortion and ensuring the camouflage's effectiveness when applied in the real world. Then we combine two renderers with different characteristics to obtain adversarial examples that are photorealistic that closely mimic real-world lighting and texture properties. The method ensures that the generated textures remain effective under diverse environmental conditions. Our adversarial camouflage can be optimized and printed in the form of 2D patterns, allowing for direct application on real vehicles. Extensive experiments demonstrated that our proposed method achieved good performance in both the digital world and the physical world.