"A good pun is its own reword": Can Large Language Models Understand Puns?

TL;DR

This paper systematically evaluates large language models' ability to understand puns through recognition, explanation, and generation tasks, introducing new metrics aligned with human cognition.

Contribution

It introduces novel evaluation methods for pun understanding in LLMs and provides insights into their capabilities and challenges.

Findings

Identifies the 'lazy pun generation' pattern in LLMs

Highlights primary challenges faced by LLMs in pun comprehension

Proposes metrics better aligned with human cognition

Abstract

Puns play a vital role in academic research due to their distinct structure and clear definition, which aid in the comprehensive analysis of linguistic humor. However, the understanding of puns in large language models (LLMs) has not been thoroughly examined, limiting their use in creative writing and humor creation. In this paper, we leverage three popular tasks, i.e., pun recognition, explanation and generation to systematically evaluate the capabilities of LLMs in pun understanding. In addition to adopting the automated evaluation metrics from prior research, we introduce new evaluation methods and metrics that are better suited to the in-context learning paradigm of LLMs. These new metrics offer a more rigorous assessment of an LLM's ability to understand puns and align more closely with human cognition than previous metrics. Our findings reveal the "lazy pun generation" pattern and identify the primary challenges LLMs encounter in understanding puns.