Can Large Language Models Outperform Non-Experts in Poetry Evaluation? A Comparative Study Using the Consensual Assessment Technique

TL;DR

This paper introduces a novel methodology adapting the Consensual Assessment Technique for Large Language Models, demonstrating that LLMs can outperform non-expert human judges in poetry evaluation with high reliability.

Contribution

The study presents a new comparative evaluation method for LLMs in poetry assessment, achieving high correlation with ground truth and surpassing non-expert human performance.

Findings

LLMs achieved a Spearman's Rank Correlation of 0.87 with ground truth.

LLMs outperformed non-expert human judges in poetry evaluation.

High inter-rater reliability was observed among LLM assessments.

Abstract

This study adapts the Consensual Assessment Technique (CAT) for Large Language Models (LLMs), introducing a novel methodology for poetry evaluation. Using a 90-poem dataset with a ground truth based on publication venue, we demonstrate that this approach allows LLMs to significantly surpass the performance of non-expert human judges. Our method, which leverages forced-choice ranking within small, randomized batches, enabled Claude-3-Opus to achieve a Spearman's Rank Correlation of 0.87 with the ground truth, dramatically outperforming the best human non-expert evaluation (SRC = 0.38). The LLM assessments also exhibited high inter-rater reliability, underscoring the methodology's robustness. These findings establish that LLMs, when guided by a comparative framework, can be effective and reliable tools for assessing poetry, paving the way for their broader application in other creative domains.