On the Wings of Imagination: Conflicting Script-based Multi-role Framework for Humor Caption Generation

TL;DR

This paper introduces HOMER, a novel multi-role LLM framework inspired by humor theory to generate creative, humorous captions for images, addressing limitations of existing methods in creativity and interpretability.

Contribution

It proposes a multi-role LLM collaboration framework guided by humor theory and retrieval augmentation, enhancing creativity and interpretability in humor caption generation.

Findings

HOMER outperforms state-of-the-art baselines on humor captioning datasets.

The framework effectively captures humor script oppositions and expands creative space.

Extensive experiments validate the superiority of HOMER in multi-modal humor generation.

Abstract

Humor is a commonly used and intricate human language in daily life. Humor generation, especially in multi-modal scenarios, is a challenging task for large language models (LLMs), which is typically as funny caption generation for images, requiring visual understanding, humor reasoning, creative imagination, and so on. Existing LLM-based approaches rely on reasoning chains or self-improvement, which suffer from limited creativity and interpretability. To address these bottlenecks, we develop a novel LLM-based humor generation mechanism based on a fundamental humor theory, GTVH. To produce funny and script-opposite captions, we introduce a humor-theory-driven multi-role LLM collaboration framework augmented with humor retrieval (HOMER). The framework consists of three LLM-based roles: (1) conflicting-script extractor that grounds humor in key script oppositions, forming the basis of caption generation; (2) retrieval-augmented hierarchical imaginator that identifies key humor targets and expands the creative space of them through diverse associations structured as imagination trees; and (3) caption generator that produces funny and diverse captions conditioned on the obtained knowledge. Extensive experiments on two New Yorker Cartoon benchmarking datasets show that HOMER outperforms state-of-the-art baselines and powerful LLM reasoning strategies on multi-modal humor captioning.