GatedCLIP: Gated Multimodal Fusion for Hateful Memes Detection

TL;DR

GatedCLIP is a novel vision-language model that improves hateful memes detection by adaptively fusing visual and textual features, achieving higher accuracy with efficient architecture.

Contribution

It introduces a gated fusion mechanism and learned projection heads to enhance CLIP's multimodal capabilities for hateful memes detection.

Findings

Achieves AUROC of 0.66 on Hateful Memes dataset

Outperforms baseline CLIP with AUROC of 0.49

Maintains computational efficiency with 350K parameters

Abstract

Detecting hateful content in multimodal memes presents unique challenges, as harmful messages often emerge from the complex interplay between benign images and text. We propose GatedCLIP, a Vision-Language model that enhances CLIP's multimodal capabilities with specialized architectural improvements for hateful memes detection. Our approach introduces learned projection heads that map CLIP embeddings to a task-optimized semantic space, a dynamic gated fusion mechanism that adaptively weights visual and textual features, and a contrastive learning objective that maintains cross-modal semantic alignment. Experiments on the Hateful Memes dataset demonstrate that GatedCLIP achieves an AUROC of 0.66, substantially outperforming the CLIP baseline (AUROC 0.49) while maintaining computational efficiency with only 350K trainable parameters.