Chumor 1.0: A Truly Funny and Challenging Chinese Humor Understanding Dataset from Ruo Zhi Ba

TL;DR

Chumor 1.0 is a novel Chinese humor dataset from Ruo Zhi Ba that challenges state-of-the-art language models and highlights the cultural nuances in humor understanding.

Contribution

This work introduces Chumor, the first Chinese humor dataset with explanations, and evaluates LLMs' ability to understand culturally specific humor.

Findings

Chumor is challenging for SOTA LLMs.

Human explanations outperform LLM-generated explanations.

The dataset captures culturally nuanced Chinese humor.

Abstract

Existing humor datasets and evaluations predominantly focus on English, lacking resources for culturally nuanced humor in non-English languages like Chinese. To address this gap, we construct Chumor, a dataset sourced from Ruo Zhi Ba (RZB), a Chinese Reddit-like platform dedicated to sharing intellectually challenging and culturally specific jokes. We annotate explanations for each joke and evaluate human explanations against two state-of-the-art LLMs, GPT-4o and ERNIE Bot, through A/B testing by native Chinese speakers. Our evaluation shows that Chumor is challenging even for SOTA LLMs, and the human explanations for Chumor jokes are significantly better than explanations generated by the LLMs.