PhonologyBench: Evaluating Phonological Skills of Large Language Models

TL;DR

This paper introduces PhonologyBench, a new benchmark to evaluate the phonological skills of large language models across tasks like grapheme-to-phoneme conversion, syllable counting, and rhyme generation, revealing notable performance gaps compared to humans.

Contribution

The paper presents PhonologyBench, the first benchmark explicitly testing LLMs' phonological abilities in English, highlighting their strengths and weaknesses without speech data.

Findings

LLMs perform notably on phonological tasks but lag behind humans.

Significant gaps of 17% and 45% in specific phonological tasks.

No single model outperforms others across all tasks.

Abstract

Phonology, the study of speech's structure and pronunciation rules, is a critical yet often overlooked component in Large Language Model (LLM) research. LLMs are widely used in various downstream applications that leverage phonology such as educational tools and poetry generation. Moreover, LLMs can potentially learn imperfect associations between orthographic and phonological forms from the training data. Thus, it is imperative to benchmark the phonological skills of LLMs. To this end, we present PhonologyBench, a novel benchmark consisting of three diagnostic tasks designed to explicitly test the phonological skills of LLMs in English: grapheme-to-phoneme conversion, syllable counting, and rhyme word generation. Despite having no access to speech data, LLMs showcased notable performance on the PhonologyBench tasks. However, we observe a significant gap of 17% and 45% on Rhyme Word Generation and Syllable counting, respectively, when compared to humans. Our findings underscore the importance of studying LLM performance on phonological tasks that inadvertently impact real-world applications. Furthermore, we encourage researchers to choose LLMs that perform well on the phonological task that is closely related to the downstream application since we find that no single model consistently outperforms the others on all the tasks.