Hate in Plain Sight: On the Risks of Moderating AI-Generated Hateful Illusions

TL;DR

This paper investigates the risks of generating and moderating hateful illusions in AI-generated images, revealing significant vulnerabilities in current moderation models and exploring potential mitigation strategies.

Contribution

It introduces the Hateful Illusion dataset and evaluates the effectiveness of existing moderation models, highlighting their limitations and proposing initial mitigation approaches.

Findings

Moderation classifiers detect less than 24.5% of hateful illusions.

Vision language models detect less than 10.2% of hateful illusions.

Current models mainly focus on surface details, missing hidden messages.

Abstract

Recent advances in text-to-image diffusion models have enabled the creation of a new form of digital art: optical illusions--visual tricks that create different perceptions of reality. However, adversaries may misuse such techniques to generate hateful illusions, which embed specific hate messages into harmless scenes and disseminate them across web communities. In this work, we take the first step toward investigating the risks of scalable hateful illusion generation and the potential for bypassing current content moderation models. Specifically, we generate 1,860 optical illusions using Stable Diffusion and ControlNet, conditioned on 62 hate messages. Of these, 1,571 are hateful illusions that successfully embed hate messages, either overtly or subtly, forming the Hateful Illusion dataset. Using this dataset, we evaluate the performance of six moderation classifiers and nine vision language models (VLMs) in identifying hateful illusions. Experimental results reveal significant vulnerabilities in existing moderation models: the detection accuracy falls below 0.245 for moderation classifiers and below 0.102 for VLMs. We further identify a critical limitation in their vision encoders, which mainly focus on surface-level image details while overlooking the secondary layer of information, i.e., hidden messages. To address this risk, we explore preliminary mitigation measures and identify the most effective approaches from the perspectives of image transformations and training-level strategies.