Labels or Input? Rethinking Augmentation in Multimodal Hate Detection

TL;DR

This paper explores how prompt optimization, fine-tuning, and automated data augmentation can significantly improve small multimodal hate detection models, making them more robust and deployable without relying on large, costly models.

Contribution

It introduces an end-to-end pipeline for prompt design and a multimodal augmentation framework that enhances small models' ability to detect implicit hate in memes.

Findings

Structured prompts and scaled supervision improve small model performance.

Counterfactually neutral meme generation reduces spurious correlations.

Prompt design and targeted augmentation narrow the gap between small and large models.

Abstract

Online hate remains a significant societal challenge, especially as multimodal content enables subtle, culturally grounded, and implicit forms of harm. Hateful memes embed hostility through text-image interactions and humor, making them difficult for automated systems to interpret. Although recent Vision-Language Models (VLMs) perform well on explicit cases, their deployment is limited by high inference costs and persistent failures on nuanced content. This work examines how far small models can be improved through prompt optimization, fine-tuning, and automated data augmentation. We introduce an end-to-end pipeline that varies prompt structure, label granularity, and training modality, showing that structured prompts and scaled supervision significantly strengthen compact VLMs. We also develop a multimodal augmentation framework that generates counterfactually neutral memes via a coordinated LLM-VLM setup, reducing spurious correlations and improving the detection of implicit hate. Ablation studies quantify the contribution of each component, demonstrating that prompt design, granular labels, and targeted augmentation collectively narrow the gap between small and large models. The results offer a practical path toward more robust and deployable multimodal hate-detection systems without relying on costly large-model inference.