Prompt-enhanced Network for Hateful Meme Classification

TL;DR

This paper introduces Pen, a prompt-enhanced network that leverages prompt learning and contrastive learning to improve hateful meme classification accuracy and generalization, addressing limitations of traditional methods.

Contribution

The paper proposes a novel prompt-enhanced network framework with contrastive learning for better hateful meme classification, outperforming existing models on public datasets.

Findings

Pen surpasses manual prompt methods in accuracy.

Pen demonstrates superior generalization capabilities.

Extensive experiments validate the effectiveness of the proposed framework.

Abstract

The dynamic expansion of social media has led to an inundation of hateful memes on media platforms, accentuating the growing need for efficient identification and removal. Acknowledging the constraints of conventional multimodal hateful meme classification, which heavily depends on external knowledge and poses the risk of including irrelevant or redundant content, we developed Pen -- a prompt-enhanced network framework based on the prompt learning approach. Specifically, after constructing the sequence through the prompt method and encoding it with a language model, we performed region information global extraction on the encoded sequence for multi-view perception. By capturing global information about inference instances and demonstrations, Pen facilitates category selection by fully leveraging sequence information. This approach significantly improves model classification accuracy. Additionally, to bolster the model's reasoning capabilities in the feature space, we introduced prompt-aware contrastive learning into the framework to improve the quality of sample feature distributions. Through extensive ablation experiments on two public datasets, we evaluate the effectiveness of the Pen framework, concurrently comparing it with state-of-the-art model baselines. Our research findings highlight that Pen surpasses manual prompt methods, showcasing superior generalization and classification accuracy in hateful meme classification tasks. Our code is available at https://github.com/juszzi/Pen.