CAHS-Attack: CLIP-Aware Heuristic Search Attack Method for Stable Diffusion

TL;DR

This paper introduces CAHS-Attack, a novel CLIP-aware heuristic search method that effectively uncovers vulnerabilities in diffusion models, revealing significant security risks in current text-to-image generation pipelines.

Contribution

The paper presents a new attack method combining MCTS and genetic algorithms to improve adversarial prompt generation without white-box access.

Findings

Achieves state-of-the-art attack success rates on diffusion models.

Identifies CLIP-based text encoders as a key vulnerability.

Demonstrates fragility of diffusion models across various prompt types.

Abstract

Diffusion models exhibit notable fragility when faced with adversarial prompts, and strengthening attack capabilities is crucial for uncovering such vulnerabilities and building more robust generative systems. Existing works often rely on white-box access to model gradients or hand-crafted prompt engineering, which is infeasible in real-world deployments due to restricted access or poor attack effect. In this paper, we propose CAHS-Attack , a CLIP-Aware Heuristic Search attack method. CAHS-Attack integrates Monte Carlo Tree Search (MCTS) to perform fine-grained suffix optimization, leveraging a constrained genetic algorithm to preselect high-potential adversarial prompts as root nodes, and retaining the most semantically disruptive outcome at each simulation rollout for efficient local search. Extensive experiments demonstrate that our method achieves state-of-the-art attack performance across both short and long prompts of varying semantics. Furthermore, we find that the fragility of SD models can be attributed to the inherent vulnerability of their CLIP-based text encoders, suggesting a fundamental security risk in current text-to-image pipelines.