Multimodal and Explainable Internet Meme Classification
Abhinav Kumar Thakur, Filip Ilievski, H\^ong-\^An Sandlin, Zhivar, Sourati, Luca Luceri, Riccardo Tommasini, Alain Mermoud

TL;DR
This paper introduces a modular, multimodal, and explainable approach to Internet meme classification, enhancing content moderation by explicitly considering meme semantics and context, and providing interpretability through example- and prototype-based reasoning.
Contribution
It presents a novel explainable architecture for meme understanding that combines multimodal models with reasoning methods, addressing limitations of black-box approaches.
Findings
Multimodal models outperform unimodal ones in detecting harmful memes.
Prototype-based reasoning provides better interpretability than example-based methods.
The system effectively identifies hate speech and misogyny in memes.
Abstract
In the current context where online platforms have been effectively weaponized in a variety of geo-political events and social issues, Internet memes make fair content moderation at scale even more difficult. Existing work on meme classification and tracking has focused on black-box methods that do not explicitly consider the semantics of the memes or the context of their creation. In this paper, we pursue a modular and explainable architecture for Internet meme understanding. We design and implement multimodal classification methods that perform example- and prototype-based reasoning over training cases, while leveraging both textual and visual SOTA models to represent the individual cases. We study the relevance of our modular and explainable models in detecting harmful memes on two existing tasks: Hate Speech Detection and Misogyny Classification. We compare the performance between…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsHate Speech and Cyberbullying Detection · Misinformation and Its Impacts · Sentiment Analysis and Opinion Mining
