Just KIDDIN: Knowledge Infusion and Distillation for Detection of INdecent Memes

TL;DR

This paper introduces a hybrid neurosymbolic framework that combines knowledge distillation from large visual language models and knowledge infusion from a large-scale commonsense knowledge graph to improve toxicity detection in hateful memes, demonstrating superior results on benchmark datasets.

Contribution

The paper presents a novel hybrid approach integrating knowledge distillation and knowledge infusion from ConceptNet into a compact visual language model for better toxicity detection.

Findings

Achieved 1.1% improvement in AU-ROC over baselines

Improved F1 score by 7%

Enhanced recall by 35% on hate speech datasets

Abstract

Toxicity identification in online multimodal environments remains a challenging task due to the complexity of contextual connections across modalities (e.g., textual and visual). In this paper, we propose a novel framework that integrates Knowledge Distillation (KD) from Large Visual Language Models (LVLMs) and knowledge infusion to enhance the performance of toxicity detection in hateful memes. Our approach extracts sub-knowledge graphs from ConceptNet, a large-scale commonsense Knowledge Graph (KG) to be infused within a compact VLM framework. The relational context between toxic phrases in captions and memes, as well as visual concepts in memes enhance the model's reasoning capabilities. Experimental results from our study on two hate speech benchmark datasets demonstrate superior performance over the state-of-the-art baselines across AU-ROC, F1, and Recall with improvements of 1.1%, 7%, and 35%, respectively. Given the contextual complexity of the toxicity detection task, our approach showcases the significance of learning from both explicit (i.e. KG) as well as implicit (i.e. LVLMs) contextual cues incorporated through a hybrid neurosymbolic approach. This is crucial for real-world applications where accurate and scalable recognition of toxic content is critical for creating safer online environments.