Do Large Language Models Grasp The Grammar? Evidence from Grammar-Book-Guided Probing in Luxembourgish

TL;DR

This paper introduces a systematic framework for evaluating grammatical understanding in large language models, using Luxembourgish as a case study, revealing that models excel in semantics but struggle with morphology and syntax.

Contribution

It proposes a novel Grammar Book Guided evaluation pipeline for systematic grammar assessment in low-resource languages like Luxembourgish.

Findings

Larger models show weak correlation between translation and grammatical understanding.

Models perform well in semantics but poorly in morphology and syntax.

Strong reasoning ability can improve grammatical comprehension.

Abstract

Grammar refers to the system of rules that governs the structural organization and the semantic relations among linguistic units such as sentences, phrases, and words within a given language. In natural language processing, there remains a notable scarcity of grammar focused evaluation protocols, a gap that is even more pronounced for low-resource languages. Moreover, the extent to which large language models genuinely comprehend grammatical structure, especially the mapping between syntactic structures and meanings, remains under debate. To investigate this issue, we propose a Grammar Book Guided evaluation pipeline intended to provide a systematic and generalizable framework for grammar evaluation consisting of four key stages, and in this work we take Luxembourgish as a case study. The results show a weak positive correlation between translation performance and grammatical understanding, indicating that strong translations do not necessarily imply deep grammatical competence. Larger models perform well overall due to their semantic strength but remain weak in morphology and syntax, struggling particularly with Minimal Pair tasks, while strong reasoning ability offers a promising way to enhance their grammatical understanding.