Truly Assessing Fluid Intelligence of Large Language Models through Dynamic Reasoning Evaluation

TL;DR

This paper introduces DRE-Bench, a hierarchical, dynamic reasoning benchmark designed to evaluate the fluid intelligence of large language models, revealing current models' limitations in high-level abstract reasoning and generalization.

Contribution

The paper presents DRE-Bench, a novel, interpretable benchmark with dynamic variants for assessing fluid intelligence in LLMs, addressing limitations of existing reasoning tests.

Findings

LLMs perform well on low-level reasoning tasks

Models struggle with high-level cognition and generalization

Current LLMs exhibit limited fluid intelligence compared to humans

Abstract

Recent advances in large language models (LLMs) have demonstrated impressive reasoning capacities that mirror human-like thinking. However, whether LLMs possess genuine fluid intelligence (i.e., the ability to reason abstractly and generalize rules in novel situations) remains an open question. Existing reasoning benchmarks either focus on domain-specific knowledge (crystallized intelligence) or lack interpretability. To address these limitations, we propose DRE-Bench, a dynamic reasoning evaluation benchmark grounded in a hierarchical cognitive framework. DRE-Bench consists of 36 abstract reasoning tasks organized across four cognitive levels, with each task featuring multiple dynamic variants that test the same underlying latent rule. This design enables fine-grained, interpretable, and reliable assessments of fluid intelligence. We evaluate a range of state-of-the-art LLMs, including both general LLMs (GPT-4o, Claude 3.7) and reasoning LLMs (o1, DeepSeek-R1, QwQ, Skywork-OR1). Experimental results reveal that although most LLMs achieve competent and robust performance in low-level cognition, they struggle with high-level cognition and exhibit limited generalization as task complexity grows. Our findings highlight the gap between current LLMs and true human-like fluid intelligence and offer a new path for systematically tracking reasoning progress in LLMs.