OneEval: Benchmarking LLM Knowledge-intensive Reasoning over Diverse Knowledge Bases

TL;DR

OneEval is a comprehensive benchmark designed to evaluate large language models' reasoning abilities across diverse structured knowledge modalities and domains, revealing persistent limitations and guiding future improvements.

Contribution

The paper introduces OneEval, a new benchmark with 4,019 instances across multiple knowledge modalities and domains, to systematically assess and analyze LLMs' structured reasoning capabilities.

Findings

LLMs show limited performance on structured reasoning tasks, with the best models achieving only 32.2% accuracy on the hardest subset.

Performance declines as the complexity of the knowledge base increases, from 53% in textual reasoning to 25% in formal logic.

Extended reasoning chains yield diminishing returns, indicating the need for models to adapt reasoning depth to task complexity.

Abstract

Large Language Models (LLMs) have demonstrated substantial progress on reasoning tasks involving unstructured text, yet their capabilities significantly deteriorate when reasoning requires integrating structured external knowledge such as knowledge graphs, code snippets, or formal logic. This limitation is partly due to the absence of benchmarks capable of systematically evaluating LLM performance across diverse structured knowledge modalities. To address this gap, we introduce \textbf{\textsc{OneEval}}, a comprehensive benchmark explicitly designed to assess the knowledge-intensive reasoning capabilities of LLMs across four structured knowledge modalities, unstructured text, knowledge graphs, code, and formal logic, and five critical domains (general knowledge, government, science, law, and programming). \textsc{OneEval} comprises 4,019 carefully curated instances and includes a challenging subset, \textsc{OneEval}\textsubscript{Hard}, consisting of 1,285 particularly difficult cases. Through extensive evaluation of 18 state-of-the-art open-source and proprietary LLMs, we establish three core findings: a) \emph{persistent limitations in structured reasoning}, with even the strongest model achieving only 32.2\% accuracy on \textsc{OneEval}\textsubscript{Hard}; b) \emph{performance consistently declines as the structural complexity of the knowledge base increases}, with accuracy dropping sharply from 53\% (textual reasoning) to 25\% (formal logic); and c) \emph{diminishing returns from extended reasoning chains}, highlighting the critical need for models to adapt reasoning depth appropriately to task complexity. We release the \textsc{OneEval} datasets, evaluation scripts, and baseline results publicly, accompanied by a leaderboard to facilitate ongoing advancements in structured knowledge reasoning.