TL;DR
POEMetric is a comprehensive framework for evaluating poetry generated by LLMs, comparing their abilities to human poets across form, creativity, emotion, and literary quality.
Contribution
This paper introduces POEMetric, the first detailed framework for assessing poetry quality and creativity in LLMs, supported by a curated dataset and extensive experiments.
Findings
Top LLMs excel in form accuracy and theme but lag in creativity and emotional resonance.
Humans outperform LLMs in overall poem quality and literary devices.
Data and code are publicly available for further research.
Abstract
Large Language Models (LLMs) can compose poetry, but how far are they from human poets? In this paper, we introduce POEMetric, the first comprehensive framework for poetry evaluation, examining 1) basic instruction-following abilities in generating poems according to a certain form and theme, 2) advanced abilities of showing creativity, lexical diversity, and idiosyncrasy, evoking emotional resonance, and using imagery and literary devices, and 3) general appraisal of the overall poem quality and estimation of authorship. We curated a human poem dataset - 203 English poems of 7 fixed forms annotated with meter, rhyme patterns and themes - and experimented with 30 LLMs for poetry generation based on the same forms and themes of the human data, totaling 6,090 LLM poems. Based on POEMetric, we assessed the performance of both human poets and LLMs through rule-based evaluation and…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
