Counting the Bugs in ChatGPT's Wugs: A Multilingual Investigation into the Morphological Capabilities of a Large Language Model

TL;DR

This study systematically evaluates ChatGPT's morphological abilities across four diverse languages using a novel wug test, revealing significant underperformance compared to specialized systems and challenging claims of human-like linguistic skills.

Contribution

First comprehensive analysis of ChatGPT's morphology across multiple languages, highlighting its limitations and providing insights into its linguistic capabilities beyond English.

Findings

ChatGPT underperforms compared to purpose-built systems.

Performance varies significantly across languages.

Claims of human-like language skills are premature.

Abstract

Large language models (LLMs) have recently reached an impressive level of linguistic capability, prompting comparisons with human language skills. However, there have been relatively few systematic inquiries into the linguistic capabilities of the latest generation of LLMs, and those studies that do exist (i) ignore the remarkable ability of humans to generalize, (ii) focus only on English, and (iii) investigate syntax or semantics and overlook other capabilities that lie at the heart of human language, like morphology. Here, we close these gaps by conducting the first rigorous analysis of the morphological capabilities of ChatGPT in four typologically varied languages (specifically, English, German, Tamil, and Turkish). We apply a version of Berko's (1958) wug test to ChatGPT, using novel, uncontaminated datasets for the four examined languages. We find that ChatGPT massively underperforms purpose-built systems, particularly in English. Overall, our results -- through the lens of morphology -- cast a new light on the linguistic capabilities of ChatGPT, suggesting that claims of human-like language skills are premature and misleading.