Multi-Modal Discussion Transformer: Integrating Text, Images and Graph Transformers to Detect Hate Speech on Social Media

TL;DR

The paper introduces mDT, a multi-modal transformer model that integrates text, images, and discussion context to improve hate speech detection on social media, outperforming comment-only methods.

Contribution

It presents a novel multi-modal transformer architecture that combines text, images, and discussion graphs for hate speech detection, along with a new dataset for evaluation.

Findings

mDT outperforms comment-only baselines in hate speech detection

The model effectively captures contextual relationships in discussions

A new dataset, HatefulDiscussions, is introduced for multi-modal analysis.

Abstract

We present the Multi-Modal Discussion Transformer (mDT), a novel methodfor detecting hate speech in online social networks such as Reddit discussions. In contrast to traditional comment-only methods, our approach to labelling a comment as hate speech involves a holistic analysis of text and images grounded in the discussion context. This is done by leveraging graph transformers to capture the contextual relationships in the discussion surrounding a comment and grounding the interwoven fusion layers that combine text and image embeddings instead of processing modalities separately. To evaluate our work, we present a new dataset, HatefulDiscussions, comprising complete multi-modal discussions from multiple online communities on Reddit. We compare the performance of our model to baselines that only process individual comments and conduct extensive ablation studies.