Benchmarking Abstract and Reasoning Abilities Through A Theoretical Perspective

TL;DR

This paper introduces a theoretically grounded benchmark with novel metrics to evaluate and analyze the abstract reasoning capabilities of large language models, revealing significant limitations and areas for improvement.

Contribution

It develops a mathematical framework and two new metrics, b3 and d, for assessing genuine abstraction versus memorization in LLMs, along with a systematic symbol remapping benchmark.

Findings

LLMs show limitations in non-decimal arithmetic and symbolic reasoning

Persistent gaps in abstraction despite chain-of-thought prompting

b3d effectively measures memory dependence and pattern recognition

Abstract

In this paper, we aim to establish a simple, effective, and theoretically grounded benchmark for rigorously probing abstract reasoning in Large Language Models (LLMs). To achieve this, we first develop a mathematic framework that defines abstract reasoning as the ability to: (i) extract essential patterns independent of surface representations, and (ii) apply consistent rules to these abstract patterns. Based on this framework, we introduce two novel complementary metrics: \(\scoreGamma\) measures basic reasoning accuracy, while \(\scoreDelta\) quantifies a model's reliance on specific symbols rather than underlying patterns - a key indicator of true abstraction versus mere memorization. To implement this measurement, we design a benchmark: systematic symbol remapping in rule-based tasks, which forces models to demonstrate genuine pattern recognition beyond superficial token matching. Extensive LLM evaluations using this benchmark (commercial API models, 7B-70B, multi-agent) reveal:1) critical limitations in non-decimal arithmetic and symbolic reasoning; 2) persistent abstraction gaps despite chain-of-thought prompting; and 3) \(\scoreDelta\)'s effectiveness in robustly measuring memory dependence by quantifying performance degradation under symbol remapping, particularly highlighting operand-specific memorization. These findings underscore that current LLMs, despite domain-specific strengths, still lack robust abstract reasoning, highlighting key areas for future improvement.