GRILE: A Benchmark for Grammar Reasoning and Explanation in Romanian LLMs

TL;DR

GRILE is a new benchmark for evaluating Romanian language models on grammar reasoning and explanations, revealing current limitations and guiding future educational NLP research in low-resource languages.

Contribution

Introduces GRILE, the first open benchmark with 1,151 Romanian exam questions to assess LLMs' answer accuracy and explanation quality, highlighting systematic weaknesses.

Findings

Gemini 2.5 Pro achieves 83% accuracy

Most open-weight models score below 65%

48% of explanations contain factual or pedagogical flaws

Abstract

LLMs (Large language models) have revolutionized NLP (Natural Language Processing), yet their pedagogical value for low-resource languages remains unclear. We present GRILE (Grammar Romanian Inference and Language Explanations) , the first open benchmark of 1,151 multiple-choice questions harvested from Romanian high-stakes exams (National Evaluation, Baccalaureate, university admissions). GRILE enables us to probe two complementary abilities of seven state-of-the-art multilingual and Romanian-specific LLMs: (i) selecting the correct answer, and (ii) producing linguistically accurate explanations. While Gemini 2.5 Pro reaches 83% accuracy, most open-weight models stay below 65%, and 48% of their explanations contain factual or pedagogical flaws according to expert review. A detailed error analysis pinpoints systematic weaknesses in morphology and in applying the latest DOOM3 orthographic norms. All data, code and a public web demo are released to catalyze future research. Our findings expose open challenges for trustworthy educational NLP in low-resource settings and establish GRILE as a new test-bed for controllable explanation generation and evaluation.