Evaluating Diversity in Automatic Poetry Generation

TL;DR

This paper assesses the diversity of automatically generated poetry, revealing current models' limitations in variety and style, and demonstrating that style-conditioning and character-level modeling improve diversity.

Contribution

It introduces a comprehensive evaluation of diversity in automatic poetry generation across multiple dimensions and compares various model types and fine-tuning methods.

Findings

Current models are underdiverse in rhyme, semantics, and length.

Style-conditioning and character-level modeling increase diversity.

Identifies key limitations for future improvement.

Abstract

Natural Language Generation (NLG), and more generally generative AI, are among the currently most impactful research fields. Creative NLG, such as automatic poetry generation, is a fascinating niche in this area. While most previous research has focused on forms of the Turing test when evaluating automatic poetry generation -- can humans distinguish between automatic and human generated poetry -- we evaluate the diversity of automatically generated poetry (with a focus on quatrains), by comparing distributions of generated poetry to distributions of human poetry along structural, lexical, semantic and stylistic dimensions, assessing different model types (word vs. character-level, general purpose LLMs vs. poetry-specific models), including the very recent LLaMA3-8B, and types of fine-tuning (conditioned vs. unconditioned). We find that current automatic poetry systems are considerably underdiverse along multiple dimensions -- they often do not rhyme sufficiently, are semantically too uniform and even do not match the length distribution of human poetry. Our experiments reveal, however, that style-conditioning and character-level modeling clearly increases diversity across virtually all dimensions we explore. Our identified limitations may serve as the basis for more genuinely diverse future poetry generation models.