Training Data Size Sensitivity in Unsupervised Rhyme Recognition

TL;DR

This paper explores how training data size impacts the accuracy of unsupervised rhyme recognition across multiple languages, using RhymeTagger and comparing it to large language models.

Contribution

It provides a comprehensive analysis of training data requirements for reliable rhyme recognition and benchmarks RhymeTagger against human agreement and LLMs.

Findings

RhymeTagger outperforms human agreement with sufficient data

LLMs struggle without phonetic representations

Training size significantly influences rhyme recognition accuracy

Abstract

Rhyme is deceptively intuitive: what is or is not a rhyme is constructed historically, scholars struggle with rhyme classification, and people disagree on whether two words are rhymed or not. This complicates automated rhymed recognition and evaluation, especially in multilingual context. This article investigates how much training data is needed for reliable unsupervised rhyme recognition using RhymeTagger, a language-independent tool that identifies rhymes based on repeating patterns in poetry corpora. We evaluate its performance across seven languages (Czech, German, English, French, Italian, Russian, and Slovene), examining how training size and language differences affect accuracy. To set a realistic performance benchmark, we assess inter-annotator agreement on a manually annotated subset of poems and analyze factors contributing to disagreement in expert annotations: phonetic similarity between rhyming words and their distance from each other in a poem. We also compare RhymeTagger to three large language models using a one-shot learning strategy. Our findings show that, once provided with sufficient training data, RhymeTagger consistently outperforms human agreement, while LLMs lacking phonetic representation significantly struggle with the task.