Human Evaluation of Creative NLG Systems: An Interdisciplinary Survey on Recent Papers
Mika H\"am\"al\"ainen, Khalid Alnajjar

TL;DR
This survey reviews human evaluation methods for creative natural language generation systems, highlighting common practices, parameters assessed, and providing guidelines for more effective and transparent future evaluations.
Contribution
It offers an interdisciplinary overview of recent evaluation practices and proposes comprehensive guidelines to improve the quality and transparency of human evaluations in creative NLG.
Findings
Most evaluations use 5-point scaled surveys.
Commonly evaluated parameters include meaning, correctness, novelty, relevance, and emotional value.
Recommendations for future evaluations include clear goal definition, multiple setups, and thorough analysis.
Abstract
We survey human evaluation in papers presenting work on creative natural language generation that have been published in INLG 2020 and ICCC 2020. The most typical human evaluation method is a scaled survey, typically on a 5 point scale, while many other less common methods exist. The most commonly evaluated parameters are meaning, syntactic correctness, novelty, relevance and emotional value, among many others. Our guidelines for future evaluation include clearly defining the goal of the generative system, asking questions as concrete as possible, testing the evaluation setup, using multiple different evaluation setups, reporting the entire evaluation process and potential biases clearly, and finally analyzing the evaluation results in a more profound way than merely reporting the most typical statistics.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsTopic Modeling · Multimodal Machine Learning Applications · Educational Games and Gamification
