A Context-aware Attention and Graph Neural Network-based Multimodal Framework for Misogyny Detection

TL;DR

This paper introduces a novel multimodal framework combining attention mechanisms, graph neural networks, and feature learning to effectively detect misogynistic content on social media, outperforming existing methods.

Contribution

It proposes a new multimodal approach with modules for attention, feature reconstruction, and content-specific learning, tailored specifically for misogyny detection.

Findings

Achieved 10.17% and 8.88% improvements in macro-F1 on two datasets.

Demonstrated effectiveness of multimodal attention and graph-based feature refinement.

Curated misogynous lexicons for enhanced textual analysis.

Abstract

A substantial portion of offensive content on social media is directed towards women. Since the approaches for general offensive content detection face a challenge in detecting misogynistic content, it requires solutions tailored to address offensive content against women. To this end, we propose a novel multimodal framework for the detection of misogynistic and sexist content. The framework comprises three modules: the Multimodal Attention module (MANM), the Graph-based Feature Reconstruction Module (GFRM), and the Content-specific Features Learning Module (CFLM). The MANM employs adaptive gating-based multimodal context-aware attention, enabling the model to focus on relevant visual and textual information and generating contextually relevant features. The GFRM module utilizes graphs to refine features within individual modalities, while the CFLM focuses on learning text and image-specific features such as toxicity features and caption features. Additionally, we curate a set of misogynous lexicons to compute the misogyny-specific lexicon score from the text. We apply test-time augmentation in feature space to better generalize the predictions on diverse inputs. The performance of the proposed approach has been evaluated on two multimodal datasets, MAMI and MMHS150K, with 11,000 and 13,494 samples, respectively. The proposed method demonstrates an average improvement of 10.17% and 8.88% in macro-F1 over existing methods on the MAMI and MMHS150K datasets, respectively.