Prompt-Guided Environmentally Consistent Adversarial Patch

TL;DR

This paper presents PG-ECAP, a novel adversarial patch generation method that uses diffusion models and alignment losses to produce natural, environmentally consistent patches capable of evading detection in physical and digital settings.

Contribution

The paper introduces a new approach combining diffusion models and alignment losses to generate environmentally consistent adversarial patches, improving naturalness and attack success.

Findings

PG-ECAP achieves higher attack success rates than existing methods.

Generated patches blend seamlessly into various environments.

The method is effective in both digital and physical attack scenarios.

Abstract

Adversarial attacks in the physical world pose a significant threat to the security of vision-based systems, such as facial recognition and autonomous driving. Existing adversarial patch methods primarily focus on improving attack performance, but they often produce patches that are easily detectable by humans and struggle to achieve environmental consistency, i.e., blending patches into the environment. This paper introduces a novel approach for generating adversarial patches, which addresses both the visual naturalness and environmental consistency of the patches. We propose Prompt-Guided Environmentally Consistent Adversarial Patch (PG-ECAP), a method that aligns the patch with the environment to ensure seamless integration into the environment. The approach leverages diffusion models to generate patches that are both environmental consistency and effective in evading detection. To further enhance the naturalness and consistency, we introduce two alignment losses: Prompt Alignment Loss and Latent Space Alignment Loss, ensuring that the generated patch maintains its adversarial properties while fitting naturally within its environment. Extensive experiments in both digital and physical domains demonstrate that PG-ECAP outperforms existing methods in attack success rate and environmental consistency.