Evaluating Relational Reasoning in LLMs with REL

TL;DR

This paper introduces REL, a benchmark framework to evaluate relational reasoning in large language models across various domains, revealing models' performance decline as relational complexity increases.

Contribution

The paper defines relational complexity as a measure of reasoning difficulty and creates REL, a new benchmark to systematically assess LLMs' higher-arity relational reasoning capabilities.

Findings

LLMs' performance decreases monotonically with increasing relational complexity.

The decline persists even with more test-time compute and in-context learning.

Current models struggle with higher-arity reasoning, highlighting a key limitation.

Abstract

Relational reasoning is the ability to infer relations that jointly bind multiple entities, attributes, or variables. This ability is central to scientific reasoning, but existing evaluations of relational reasoning in large language models often focus on structured inputs such as tables, graphs, or synthetic tasks, and do not isolate the difficulty introduced by higher-arity relational binding. We study this problem through the lens of Relational Complexity (RC), which we define as the minimum number of independent entities or operands that must be simultaneously bound to apply a relation. RC provides a principled way to vary reasoning difficulty while controlling for confounders such as input size, vocabulary, and representational choices. Building on RC, we introduce REL, a generative benchmark framework spanning algebra, chemistry, and biology that varies RC within each domain. Across frontier LLMs, performance degrades consistently and monotonically as RC increases, even when the total number of entities is held fixed. This failure mode persists with increased test-time compute and in-context learning, suggesting a limitation tied to the arity of the required relational binding rather than to insufficient inference steps or lack of exposure to examples. Our results identify a regime of higher-arity reasoning in which current models struggle, and motivate re-examining benchmarks through the lens of relational complexity.