Beyond Meme Templates: Limitations of Visual Similarity Measures in Meme Matching

TL;DR

This paper investigates the limitations of current visual similarity measures in meme matching, especially for non-template memes, and explores new approaches including segment-wise similarity and prompting-based methods.

Contribution

It introduces a broader formulation of meme matching beyond template-based methods and evaluates novel similarity measures and multimodal prompting approaches.

Findings

Segment-wise similarity measures outperform whole-image measures on non-template memes.

Conventional similarity measures excel at template-based meme matching.

Accurately matching non-template memes remains an open challenge.

Abstract

Internet memes, now a staple of digital communication, play a pivotal role in how users engage within online communities and allow researchers to gain insight into contemporary digital culture. These engaging user-generated content are characterised by their reuse of visual elements also found in other memes. Matching instances of memes via these shared visual elements, called Meme Matching, is the basis of a wealth of meme analysis approaches. However, most existing methods assume that every meme consists of a shared visual background, called a Template, with some overlaid text, thereby limiting meme matching to comparing the background image alone. Current approaches exclude the many memes that are not template-based and limit the effectiveness of automated meme analysis and would not be effective at linking memes to contemporary web-based meme dictionaries. In this work, we introduce a broader formulation of meme matching that extends beyond template matching. We show that conventional similarity measures, including a novel segment-wise computation of the similarity measures, excel at matching template-based memes but fall short when applied to non-template-based meme formats. However, the segment-wise approach was found to consistently outperform the whole-image measures on matching non-template-based memes. Finally, we explore a prompting-based approach using a pretrained Multimodal Large Language Model for meme matching. Our results highlight that accurately matching memes via shared visual elements, not just background templates, remains an open challenge that requires more sophisticated matching techniques.