Large Vision-Language Models for Knowledge-Grounded Data Annotation of Memes

TL;DR

This paper introduces a large meme dataset, an automated annotation pipeline using vision-language models, and a specialized CLIP model for meme-text retrieval to improve understanding and analysis of memes.

Contribution

It presents a new large-scale meme dataset, an automated annotation framework, and a fine-tuned CLIP model for meme-text retrieval, advancing meme analysis capabilities.

Findings

Created the CM50 meme dataset with 33,000 memes

Developed an automated annotation pipeline for memes

Enhanced meme-text retrieval with a specialized CLIP model

Abstract

Memes have emerged as a powerful form of communication, integrating visual and textual elements to convey humor, satire, and cultural messages. Existing research has focused primarily on aspects such as emotion classification, meme generation, propagation, interpretation, figurative language, and sociolinguistics, but has often overlooked deeper meme comprehension and meme-text retrieval. To address these gaps, this study introduces ClassicMemes-50-templates (CM50), a large-scale dataset consisting of over 33,000 memes, centered around 50 popular meme templates. We also present an automated knowledge-grounded annotation pipeline leveraging large vision-language models to produce high-quality image captions, meme captions, and literary device labels overcoming the labor intensive demands of manual annotation. Additionally, we propose a meme-text retrieval CLIP model (mtrCLIP) that utilizes cross-modal embedding to enhance meme analysis, significantly improving retrieval performance. Our contributions include:(1) a novel dataset for large-scale meme study, (2) a scalable meme annotation framework, and (3) a fine-tuned CLIP for meme-text retrieval, all aimed at advancing the understanding and analysis of memes at scale.