IOLBENCH: Benchmarking LLMs on Linguistic Reasoning

TL;DR

This paper introduces IOLBENCH, a new benchmark based on linguistics Olympiad problems, to evaluate and analyze the reasoning capabilities of large language models in linguistic tasks, revealing their current limitations.

Contribution

The paper presents IOLBENCH, a novel dataset for testing LLMs on diverse linguistic reasoning tasks, and provides comprehensive benchmarking results highlighting models' strengths and weaknesses.

Findings

LLMs struggle with complex linguistic reasoning tasks.

Models show limited ability in rule abstraction and compositional generalization.

Benchmark reveals significant gaps in current models' linguistic reasoning abilities.

Abstract

Despite the remarkable advancements and widespread applications of deep neural networks, their ability to perform reasoning tasks remains limited, particularly in domains requiring structured, abstract thought. In this paper, we investigate the linguistic reasoning capabilities of state-of-the-art large language models (LLMs) by introducing IOLBENCH, a novel benchmark derived from International Linguistics Olympiad (IOL) problems. This dataset encompasses diverse problems testing syntax, morphology, phonology, and semantics, all carefully designed to be self-contained and independent of external knowledge. These tasks challenge models to engage in metacognitive linguistic reasoning, requiring the deduction of linguistic rules and patterns from minimal examples. Through extensive benchmarking of leading LLMs, we find that even the most advanced models struggle to handle the intricacies of linguistic complexity, particularly in areas demanding compositional generalization and rule abstraction. Our analysis highlights both the strengths and persistent limitations of current models in linguistic problem-solving, offering valuable insights into their reasoning capabilities. By introducing IOLBENCH, we aim to foster further research into developing models capable of human-like reasoning, with broader implications for the fields of computational linguistics and artificial intelligence.