Sonnet or Not, Bot? Poetry Evaluation for Large Models and Datasets

TL;DR

This paper introduces a benchmark dataset and evaluation method for assessing large language models' ability to recognize poetic forms, revealing high accuracy for fixed forms and challenges with unfixed forms, and analyzing memorization effects.

Contribution

It provides a new dataset and evaluation framework for poetic form recognition in LLMs, highlighting their strengths and limitations across different poetic structures.

Findings

LLMs can identify fixed poetic forms with high accuracy

Performance drops on unfixed and topic-based poetic forms

Memorization may influence model performance on poetry recognition

Abstract

Large language models (LLMs) can now generate and recognize poetry. But what do LLMs really know about poetry? We develop a task to evaluate how well LLMs recognize one aspect of English-language poetry--poetic form--which captures many different poetic features, including rhyme scheme, meter, and word or line repetition. By using a benchmark dataset of over 4.1k human expert-annotated poems, we show that state-of-the-art LLMs can successfully identify both common and uncommon fixed poetic forms--such as sonnets, sestinas, and pantoums--with surprisingly high accuracy. However, performance varies significantly by poetic form; the models struggle to identify unfixed poetic forms, especially those based on topic or visual features. We additionally measure how many poems from our benchmark dataset are present in popular pretraining datasets or memorized by GPT-4, finding that pretraining presence and memorization may improve performance on this task, but results are inconclusive. We release a benchmark evaluation dataset with 1.4k public domain poems and form annotations, results of memorization experiments and data audits, and code.