Cracking the Code of Juxtaposition: Can AI Models Understand the Humorous Contradictions

TL;DR

This paper introduces the YesBut benchmark to evaluate AI's ability to understand humorous contradictions in comics, revealing current models' limitations compared to humans.

Contribution

It presents a new benchmark for assessing AI comprehension of humorous narratives involving nonlinear and contradictory content.

Findings

State-of-the-art models underperform humans in understanding humorous comics.

The benchmark covers tasks from literal comprehension to deep narrative reasoning.

Current models show significant room for improvement in grasping humor nuances.

Abstract

Recent advancements in large multimodal language models have demonstrated remarkable proficiency across a wide range of tasks. Yet, these models still struggle with understanding the nuances of human humor through juxtaposition, particularly when it involves nonlinear narratives that underpin many jokes and humor cues. This paper investigates this challenge by focusing on comics with contradictory narratives, where each comic consists of two panels that create a humorous contradiction. We introduce the YesBut benchmark, which comprises tasks of varying difficulty aimed at assessing AI's capabilities in recognizing and interpreting these comics, ranging from literal content comprehension to deep narrative reasoning. Through extensive experimentation and analysis of recent commercial or open-sourced large (vision) language models, we assess their capability to comprehend the complex interplay of the narrative humor inherent in these comics. Our results show that even state-of-the-art models still lag behind human performance on this task. Our findings offer insights into the current limitations and potential improvements for AI in understanding human creative expressions.