MemeIntel: Explainable Detection of Propagandistic and Hateful Memes

TL;DR

This paper introduces MemeXplain, a large-scale dataset and a multi-stage training approach for detecting propagandistic and hateful memes, enhancing explainability and accuracy in multimodal content moderation.

Contribution

It presents the first large-scale dataset for explainable detection of propagandistic and hateful memes and proposes a multi-stage optimization method for improved performance.

Findings

Significant accuracy improvements over state-of-the-art models.

Effective joint modeling of label detection and explanation generation.

Public release of MemeXplain dataset and tools.

Abstract

The proliferation of multimodal content on social media presents significant challenges in understanding and moderating complex, context-dependent issues such as misinformation, hate speech, and propaganda. While efforts have been made to develop resources and propose new methods for automatic detection, limited attention has been given to jointly modeling label detection and the generation of explanation-based rationales, which often leads to degraded classification performance when trained simultaneously. To address this challenge, we introduce MemeXplain, an explanation-enhanced dataset for propagandistic memes in Arabic and hateful memes in English, making it the first large-scale resource for these tasks. To solve these tasks, we propose a multi-stage optimization approach and train Vision-Language Models (VLMs). Our results show that this strategy significantly improves both label detection and explanation generation quality over the base model, outperforming the current state-of-the-art with an absolute improvement of ~1.4% (Acc) on ArMeme and ~2.2% (Acc) on Hateful Memes. For reproducibility and future research, we aim to make the MemeXplain dataset and scripts publicly available (https://github.com/MohamedBayan/MemeIntel).