PunchBench: Benchmarking MLLMs in Multimodal Punchline Comprehension

TL;DR

This paper introduces PunchBench, a comprehensive benchmark for evaluating multimodal large language models' ability to understand humor and sarcasm in image-caption pairs, addressing existing limitations and proposing a new improvement strategy.

Contribution

It presents PunchBench, a novel benchmark with diverse questions and domain coverage, and introduces SC-CoQ, a strategy to improve punchline comprehension in MLLMs.

Findings

Significant gap between MLLMs and humans in punchline comprehension.

SC-CoQ strategy improves MLLMs' performance on PunchBench.

Enhanced evaluation accuracy by generating synonymous and antonymous captions.

Abstract

Multimodal punchlines, which involve humor or sarcasm conveyed in image-caption pairs, are a popular way of communication on online multimedia platforms. With the rapid development of multimodal large language models (MLLMs), it is essential to assess their ability to effectively comprehend these punchlines. However, existing benchmarks on punchline comprehension suffer from three major limitations: 1) language shortcuts that allow models to solely rely on text, 2) lack of question diversity, and 3) narrow focus on a specific domain of multimodal content (e.g., cartoon). To address these limitations, we introduce a multimodal \textbf{Punch}line comprehension \textbf{Bench}mark, named \textbf{PunchBench}, which is tailored for accurate and comprehensive evaluation of punchline comprehension. To enhance the evaluation accuracy, we generate synonymous and antonymous captions by modifying original captions, which mitigates the impact of shortcuts in the captions. To provide a comprehensive evaluation, PunchBench incorporates diverse question formats and image-captions from various domains. On this basis, we conduct extensive evaluations and reveal a significant gap between state-of-the-art MLLMs and humans in punchline comprehension. To improve punchline comprehension, we propose Simple-to-Complex Chain-of-Question (SC-CoQ) strategy, enabling the models to incrementally address complicated questions by first mastering simple ones. SC-CoQ effectively enhances the performance of various MLLMs on PunchBench, surpassing in-context learning and chain-of-thought.