"I See What You Did There": Can Large Vision-Language Models Understand Multimodal Puns?

TL;DR

This paper introduces MultiPun, a dataset and evaluation framework to assess vision-language models' ability to understand multimodal puns, revealing current limitations and proposing strategies for improvement.

Contribution

The paper presents the first systematic study of multimodal pun understanding in VLMs, including a new dataset, evaluation methods, and strategies to enhance pun comprehension.

Findings

Most models struggle to distinguish genuine puns from distractors.

Prompt and model-level strategies improve pun understanding by 16.5% in F1 scores.

MultiPun dataset enables rigorous benchmarking of multimodal pun comprehension.

Abstract

Puns are a common form of rhetorical wordplay that exploits polysemy and phonetic similarity to create humor. In multimodal puns, visual and textual elements synergize to ground the literal sense and evoke the figurative meaning simultaneously. Although Vision-Language Models (VLMs) are widely used in multimodal understanding and generation, their ability to understand puns has not been systematically studied due to a scarcity of rigorous benchmarks. To address this, we first propose a multimodal pun generation pipeline. We then introduce MultiPun, a dataset comprising diverse types of puns alongside adversarial non-pun distractors. Our evaluation reveals that most models struggle to distinguish genuine puns from these distractors. Moreover, we propose both prompt-level and model-level strategies to enhance pun comprehension, with an average improvement of 16.5% in F1 scores. Our findings provide valuable insights for developing future VLMs that master the subtleties of human-like humor via cross-modal reasoning.