Targeted Syntactic Evaluation of Language Models on Georgian Case Alignment

TL;DR

This study assesses transformer-based language models' ability to handle Georgian's rare split-ergative case system, revealing challenges in ergative case assignment and highlighting data scarcity issues in low-resource languages.

Contribution

Introduces a novel dataset and methodology for syntactic evaluation of Georgian language models, focusing on split-ergative case alignment and providing insights into low-resource language modeling.

Findings

Models perform poorly on ergative case assignment.

Performance correlates with frequency of noun forms (NOM > DAT > ERG).

Data scarcity impacts model accuracy on ergative cases.

Abstract

This paper evaluates the performance of transformer-based language models on split-ergative case alignment in Georgian, a particularly rare system for assigning grammatical cases to mark argument roles. We focus on subject and object marking determined through various permutations of nominative, ergative, and dative noun forms. A treebank-based approach for the generation of minimal pairs using the Grew query language is implemented. We create a dataset of 370 syntactic tests made up of seven tasks containing 50-70 samples each, where three noun forms are tested in any given sample. Five encoder- and two decoder-only models are evaluated with word- and/or sentence-level accuracy metrics. Regardless of the specific syntactic makeup, models performed worst in assigning the ergative case correctly and strongest in assigning the nominative case correctly. Performance correlated with the overall frequency distribution of the three forms (NOM > DAT > ERG). Though data scarcity is a known issue for low-resource languages, we show that the highly specific role of the ergative along with a lack of available training data likely contributes to poor performance on this case. The dataset is made publicly available and the methodology provides an interesting avenue for future syntactic evaluations of languages where benchmarks are limited.