STEMVerse: A Dual-Axis Diagnostic Framework for STEM Reasoning in Large Language Models

TL;DR

STEMVerse is a diagnostic framework that systematically evaluates large language models' STEM reasoning by analyzing their performance across academic disciplines and cognitive complexities, revealing structural failure patterns.

Contribution

It introduces a dual-axis diagnostic approach that maps LLM performance in STEM across discipline and cognition, offering detailed insights beyond aggregate scores.

Findings

Re-aggregated 20,000+ STEM problems into a unified capability space

Identified structural failure patterns in LLM STEM reasoning

Provided actionable insights for model improvement

Abstract

As Large Language Models (LLMs) achieve significant breakthroughs in complex reasoning tasks, evaluating their proficiency in science, technology, engineering, and mathematics (STEM) has become a primary method for measuring machine intelligence. However, current evaluation paradigms often treat benchmarks as isolated "silos," offering only monolithic aggregate scores that neglect the intricacies of both academic specialization and cognitive depth. This result-oriented approach fails to distinguish whether model errors stem from insufficient domain knowledge or deficiencies in cognitive capacity, thereby limiting the diagnostic value. To address this, we propose STEMVerse, a diagnostic framework designed to systematically analyze the STEM reasoning capabilities of LLMs. This framework characterizes model performance across academic specialization and cognitive complexity to map the capability required for reasoning. We re-aggregate over 20,000 STEM problems from mainstream benchmarks into a unified "Discipline $\times$ Cognition" capability space, assigning dual-axis labels to every instance. Utilizing this unified diagnostic framework, we systematically evaluate representative LLM families across varying parameter scales and training paradigms. Our empirical results reveal structural failure patterns in STEM reasoning. By integrating multi-disciplinary coverage and fine-grained cognitive stratification into a unified framework, STEMVerse provides a clear and actionable perspective for understanding the scientific reasoning characteristics of LLMs.