UNVEILING: What Makes Linguistics Olympiad Puzzles Tricky for LLMs?

TL;DR

This paper investigates why large language models struggle with linguistics puzzles from Olympiads, revealing that morphological complexity and language-specific features significantly impact their reasoning abilities, and suggesting improved tokenization methods.

Contribution

It provides a detailed analysis of LLM performance on low-resource language puzzles, identifying key linguistic features affecting reasoning and proposing targeted preprocessing improvements.

Findings

LLMs perform poorly on morphologically complex puzzles

Performance improves with language-specific tokenization

Certain linguistic features are easier for LLMs to handle

Abstract

Large language models (LLMs) have demonstrated potential in reasoning tasks, but their performance on linguistics puzzles remains consistently poor. These puzzles, often derived from Linguistics Olympiad (LO) contests, provide a minimal contamination environment to assess LLMs' linguistic reasoning abilities across low-resource languages. This work analyses LLMs' performance on 629 problems across 41 low-resource languages by labelling each with linguistically informed features to unveil weaknesses. Our analyses show that LLMs struggle with puzzles involving higher morphological complexity and perform better on puzzles involving linguistic features that are also found in English. We also show that splitting words into morphemes as a pre-processing step improves solvability, indicating a need for more informed and language-specific tokenisers. These findings thus offer insights into some challenges in linguistic reasoning and modelling of low-resource languages.