LingGym: How Far Are LLMs from Thinking Like Field Linguists?

TL;DR

This paper presents LingGym, a benchmark for evaluating LLMs' ability to perform meta-linguistic reasoning across diverse languages using structured linguistic data, revealing both progress and limitations in linguistic inference capabilities.

Contribution

Introduces LingGym, a novel benchmark for assessing LLMs' generalization in linguistic reasoning across typologically diverse languages using structured linguistic cues.

Findings

Structured linguistic cues improve reasoning performance.

LLMs show promise but have limitations in low-resource language inference.

Performance varies across models and linguistic structures.

Abstract

This paper introduces LingGym, a new benchmark that evaluates LLMs' capacity for meta-linguistic reasoning using Interlinear Glossed Text (IGT) and grammatical descriptions extracted from 18 typologically diverse reference grammars. Unlike previous work that focuses on specific downstream tasks, we assess whether LLMs can generalize linguistic inference across low-resource languages and structures not seen during training. We present a controlled evaluation task: Word-Gloss Inference, in which the model must infer a missing word and gloss from context using varying levels of linguistic information (e.g., glosses, grammatical explanations, translations). Our results show that incorporating structured linguistic cues leads to consistent improvements in reasoning performance across all models. This work highlights both the promise and current limitations of using LLMs for typologically informed linguistic analysis and low-resource language documentation.