The Iron(ic) Melting Pot: Reviewing Human Evaluation in Humour, Irony and Sarcasm Generation

TL;DR

This paper emphasizes the importance of transparent and demographic-aware human evaluation in generating humor, irony, and sarcasm in natural language processing, highlighting current shortcomings in reporting and methodology.

Contribution

It advocates for detailed demographic reporting in evaluations of esoteric language forms and analyzes how participant variables influence interpretation.

Findings

Evaluation reports often lack demographic details

Crowdsourcing is frequently used for evaluator recruitment

Participant demographics significantly affect interpretation

Abstract

Human evaluation is often considered to be the gold standard method of evaluating a Natural Language Generation system. However, whilst its importance is accepted by the community at large, the quality of its execution is often brought into question. In this position paper, we argue that the generation of more esoteric forms of language - humour, irony and sarcasm - constitutes a subdomain where the characteristics of selected evaluator panels are of utmost importance, and every effort should be made to report demographic characteristics wherever possible, in the interest of transparency and replicability. We support these claims with an overview of each language form and an analysis of examples in terms of how their interpretation is affected by different participant variables. We additionally perform a critical survey of recent works in NLG to assess how well evaluation procedures are reported in this subdomain, and note a severe lack of open reporting of evaluator demographic information, and a significant reliance on crowdsourcing platforms for recruitment.