Rainbow Noise: Stress-Testing Multimodal Harmful-Meme Detectors on LGBTQ Content

TL;DR

This paper introduces a robustness benchmark for detecting LGBTQ+ harmful memes, evaluates two state-of-the-art detectors, and proposes a lightweight Text Denoising Adapter to improve their resilience against caption and image attacks.

Contribution

It creates the first comprehensive robustness benchmark for multimodal harmful-meme detection and demonstrates that a lightweight TDA significantly enhances model robustness.

Findings

MemeCLIP degrades more slowly under attacks.

MemeBLIP2 is highly sensitive to caption edits.

The TDA improves robustness of MemeBLIP2 significantly.

Abstract

Hateful memes aimed at LGBTQ\,+ communities often evade detection by tweaking either the caption, the image, or both. We build the first robustness benchmark for this setting, pairing four realistic caption attacks with three canonical image corruptions and testing all combinations on the PrideMM dataset. Two state-of-the-art detectors, MemeCLIP and MemeBLIP2, serve as case studies, and we introduce a lightweight \textbf{Text Denoising Adapter (TDA)} to enhance the latter's resilience. Across the grid, MemeCLIP degrades more gently, while MemeBLIP2 is particularly sensitive to the caption edits that disrupt its language processing. However, the addition of the TDA not only remedies this weakness but makes MemeBLIP2 the most robust model overall. Ablations reveal that all systems lean heavily on text, but architectural choices and pre-training data significantly impact robustness. Our benchmark exposes where current multimodal safety models crack and demonstrates that targeted, lightweight modules like the TDA offer a powerful path towards stronger defences.