TurBLiMP: A Turkish Benchmark of Linguistic Minimal Pairs

TL;DR

TurBLiMP is a comprehensive Turkish benchmark assessing linguistic minimal pairs across 16 phenomena, revealing that advanced language models still struggle with Turkish-specific grammatical complexities like word order and morphology.

Contribution

This paper introduces TurBLiMP, the first Turkish benchmark for linguistic minimal pairs, focusing on underexplored properties like word order flexibility and morphological subordination.

Findings

Large language models struggle with Turkish grammatical phenomena.

Models show different sensitivities to word order and morphology compared to humans.

TurBLiMP fills a critical gap in Turkish linguistic evaluation resources.

Abstract

We introduce TurBLiMP, the first Turkish benchmark of linguistic minimal pairs, designed to evaluate the linguistic abilities of monolingual and multilingual language models (LMs). Covering 16 linguistic phenomena with 1000 minimal pairs each, TurBLiMP fills an important gap in linguistic evaluation resources for Turkish. In designing the benchmark, we give extra attention to two properties of Turkish that remain understudied in current syntactic evaluations of LMs, namely word order flexibility and subordination through morphological processes. Our experiments on a wide range of LMs and a newly collected set of human acceptability judgments reveal that even cutting-edge Large LMs still struggle with grammatical phenomena that are not challenging for humans, and may also exhibit different sensitivities to word order and morphological complexity compared to humans.