P-CoT: A Pedagogically-motivated Participatory Chain-of-Thought Prompting for Phonological Reasoning in LLMs

TL;DR

This paper introduces P-CoT, a pedagogically-inspired prompting method that significantly improves phonological reasoning in large language models, surpassing previous methods and even human performance on certain tasks.

Contribution

The paper presents a novel P-CoT prompting technique based on educational theories, enhancing phonological reasoning in LLMs beyond traditional few-shot learning.

Findings

P-CoT consistently improves model performance up to 52%.

It surpasses human baselines in some phonological tasks.

Few-shot learning shows inconsistent gains.

Abstract

This study explores the potential of phonological reasoning within text-based large language models (LLMs). Utilizing the PhonologyBench benchmark, we assess tasks like rhyme word generation, g2p conversion, and syllable counting. Our evaluations across 12 LLMs reveal that while few-shot learning offers inconsistent gains, the introduction of a novel Pedagogically-motivated Participatory Chain-of-Thought (P-CoT) prompt, which is anchored in educational theories like scaffolding and discovery learning, consistently enhances performance. This method leverages structured guidance to activate latent phonological abilities, achieving up to 52% improvement and even surpassing human baselines in certain tasks. Future work could aim to optimize P-CoT prompts for specific models or explore their application across different linguistic domains.