Seeing Through VisualBERT: A Causal Adventure on Memetic Landscapes
Dibyanayan Bandyopadhyay, Mohammed Hasanuzzaman, Asif Ekbal

TL;DR
This paper introduces a causal framework using Structural Causal Models to interpret VisualBERT's decisions in offensive meme detection, highlighting the limitations of attribution methods and improving transparency.
Contribution
It proposes a novel causal interpretability framework for VisualBERT, incorporating de-confounding, adversarial learning, and dynamic routing to enhance understanding of model behavior.
Findings
Input attribution methods do not ensure causality in this framework.
The proposed SCM-based approach improves interpretability of model decisions.
Quantitative analysis shows the effectiveness of modeling choices like de-confounding.
Abstract
Detecting offensive memes is crucial, yet standard deep neural network systems often remain opaque. Various input attribution-based methods attempt to interpret their behavior, but they face challenges with implicitly offensive memes and non-causal attributions. To address these issues, we propose a framework based on a Structural Causal Model (SCM). In this framework, VisualBERT is trained to predict the class of an input meme based on both meme input and causal concepts, allowing for transparent interpretation. Our qualitative evaluation demonstrates the framework's effectiveness in understanding model behavior, particularly in determining whether the model was right due to the right reason, and in identifying reasons behind misclassification. Additionally, quantitative analysis assesses the significance of proposed modelling choices, such as de-confounding, adversarial learning, and…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsDigital Games and Media · Media, Communication, and Education · Design Education and Practice
MethodsVisualBERT
