Emotion-Aware Multimodal Fusion for Meme Emotion Detection
Shivam Sharma, Ramaneswaran S, Md. Shad Akhtar, Tanmoy Chakraborty

TL;DR
This paper introduces a new multimodal neural framework called ALFRED for detecting emotions in memes, leveraging a novel dataset and explicit modeling of visual cues, achieving superior performance and demonstrating strong generalizability across datasets.
Contribution
The paper presents ALFRED, a novel emotion-aware multimodal fusion model for meme emotion detection, and introduces MOOD, a new dataset with six basic emotions, addressing limitations of prior approaches.
Findings
ALFRED outperforms existing baselines by 4.94% F1 score.
ALFRED achieves strong results on Memotion, HarMeme, and Dank Memes datasets.
ALFRED's interpretability is analyzed through attention maps.
Abstract
The ever-evolving social media discourse has witnessed an overwhelming use of memes to express opinions or dissent. Besides being misused for spreading malcontent, they are mined by corporations and political parties to glean the public's opinion. Therefore, memes predominantly offer affect-enriched insights towards ascertaining the societal psyche. However, the current approaches are yet to model the affective dimensions expressed in memes effectively. They rely extensively on large multimodal datasets for pre-training and do not generalize well due to constrained visual-linguistic grounding. In this paper, we introduce MOOD (Meme emOtiOns Dataset), which embodies six basic emotions. We then present ALFRED (emotion-Aware muLtimodal Fusion foR Emotion Detection), a novel multimodal neural framework that (i) explicitly models emotion-enriched visual cues, and (ii) employs an efficient…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsEmotion and Mood Recognition
