Grounded Satirical Generation with RAG

TL;DR

This paper introduces a novel RAG-based pipeline for grounded satire generation using news data, along with a new evaluation framework and annotated dataset, revealing insights into political relevance and humor perception.

Contribution

It presents a new RAG-based method for satire generation, a task-specific evaluation framework, and annotated dataset, advancing research in humor and satire generation.

Findings

Generated definitions are perceived as more political than humorous.

Topic-based word selection and RAG improve political relevance.

LLMs correlate with human judgments on political relevance but not on humor.

Abstract

Humor generation remains challenging task for Large Language Models (LLMs), due to their subjective nature. We focus on satire, a form of humor strongly shaped by context. In this work, we present a novel pipeline for grounded satire generation that uses Retrieval-Augmented Generation (RAG) over current news to produce satirical dictionary definitions in the Finnish context. We also introduce a new task-specific evaluation framework and annotate 100 generated definitions with six human annotators, enabling analysis across multiple experimental conditions, including cultural background, source-word type, and the presence or absence of RAG. Our results show that the generated definitions are perceived as more political than humorous. Both topic-based word selection and RAG improve the political relevance of the outputs, but neither yields clear gains in humor generation. In addition, our LLM-as-a-judge evaluation of five state-of-the-art models indicates that LLMs correlate well with human judgments on political relevance, but perform poorly on humor. We release our code and annotated dataset to support further research on grounded satire generation and evaluation.