Hateful Meme Detection through Context-Sensitive Prompting and Fine-Grained Labeling

TL;DR

This paper introduces an end-to-end framework for detecting hateful memes by integrating multi-modal prompting and fine-grained labeling, significantly improving accuracy and AUROC in complex social media content moderation tasks.

Contribution

It presents a novel end-to-end optimization pipeline that jointly considers modalities, prompting, labeling, and fine-tuning for improved multi-modal classification.

Findings

Achieved highest accuracy and AUROC in hateful meme detection

End-to-end optimization outperforms isolated approaches

Ablation studies confirm the importance of integrated optimization

Abstract

The prevalence of multi-modal content on social media complicates automated moderation strategies. This calls for an enhancement in multi-modal classification and a deeper understanding of understated meanings in images and memes. Although previous efforts have aimed at improving model performance through fine-tuning, few have explored an end-to-end optimization pipeline that accounts for modalities, prompting, labeling, and fine-tuning. In this study, we propose an end-to-end conceptual framework for model optimization in complex tasks. Experiments support the efficacy of this traditional yet novel framework, achieving the highest accuracy and AUROC. Ablation experiments demonstrate that isolated optimizations are not ineffective on their own.