IMPACT: Inflectional Morphology Probes Across Complex Typologies

TL;DR

This paper introduces IMPACT, a synthetic evaluation framework to assess multilingual LLMs' understanding of complex inflectional morphology across five rich languages, revealing significant performance gaps and areas for improvement.

Contribution

The paper presents a novel, publicly available framework for evaluating LLMs on complex morphological phenomena across multiple languages, highlighting their limitations.

Findings

LLMs perform poorly on complex morphological patterns

Chain of Thought can degrade LLM performance on morphology tasks

Significant gaps exist in LLMs' understanding of linguistic complexity

Abstract

Large Language Models (LLMs) have shown significant progress on various multilingual benchmarks and are increasingly used to generate and evaluate text in non-English languages. However, while they may produce fluent outputs, it remains unclear to what extent these models truly grasp the underlying linguistic complexity of those languages, particularly in morphology. To investigate this, we introduce IMPACT, a synthetically generated evaluation framework focused on inflectional morphology, which we publicly release, designed to evaluate LLM performance across five morphologically rich languages: Arabic, Russian, Finnish, Turkish, and Hebrew. IMPACT includes unit-test-style cases covering both shared and language-specific phenomena, from basic verb inflections (e.g., tense, number, gender) to unique features like Arabic's reverse gender agreement and vowel harmony in Finnish and Turkish. We assess eight multilingual LLMs that, despite strong English performance, struggle with other languages and uncommon morphological patterns, especially when judging ungrammatical examples. We also show that Chain of Thought and Thinking Models can degrade performance. Our work exposes gaps in LLMs' handling of linguistic complexity, pointing to clear room for improvement. To support further research, we publicly release the IMPACT framework.