M3Hop-CoT: Misogynous Meme Identification with Multimodal Multi-hop Chain-of-Thought

TL;DR

This paper introduces M3Hop-CoT, a multimodal multi-hop chain-of-thought framework that improves misogynous meme detection by integrating visual and textual cues, emotional context, and cultural knowledge, validated on multiple datasets.

Contribution

The paper proposes a novel multimodal multi-hop CoT framework combining CLIP and reasoning modules, addressing cultural and emotional cues for misogynous meme detection, outperforming existing methods.

Findings

Achieves high macro-F1 score on SemEval-2022 MAMI dataset.

Demonstrates strong generalization across various meme datasets.

Effectively incorporates emotion and context in multimodal reasoning.

Abstract

In recent years, there has been a significant rise in the phenomenon of hate against women on social media platforms, particularly through the use of misogynous memes. These memes often target women with subtle and obscure cues, making their detection a challenging task for automated systems. Recently, Large Language Models (LLMs) have shown promising results in reasoning using Chain-of-Thought (CoT) prompting to generate the intermediate reasoning chains as the rationale to facilitate multimodal tasks, but often neglect cultural diversity and key aspects like emotion and contextual knowledge hidden in the visual modalities. To address this gap, we introduce a Multimodal Multi-hop CoT (M3Hop-CoT) framework for Misogynous meme identification, combining a CLIP-based classifier and a multimodal CoT module with entity-object-relationship integration. M3Hop-CoT employs a three-step multimodal prompting principle to induce emotions, target awareness, and contextual knowledge for meme analysis. Our empirical evaluation, including both qualitative and quantitative analysis, validates the efficacy of the M3Hop-CoT framework on the SemEval-2022 Task 5 (MAMI task) dataset, highlighting its strong performance in the macro-F1 score. Furthermore, we evaluate the model's generalizability by evaluating it on various benchmark meme datasets, offering a thorough insight into the effectiveness of our approach across different datasets.