STEM: Efficient Relative Capability Evaluation of LLMs through Structured Transition Samples

TL;DR

STEM is a lightweight, interpretable evaluation method that efficiently estimates the relative capabilities of large language models by analyzing significant transition samples across multiple benchmarks.

Contribution

This paper introduces STEM, a novel evaluation framework that uses structured transition samples to accurately and efficiently compare LLM capabilities without extensive full evaluations.

Findings

STEM reliably captures performance trends across models.

It aligns well with ground-truth rankings of model capabilities.

STEM is scalable and architecture-agnostic.

Abstract

Evaluating large language models (LLMs) has become increasingly challenging as model capabilities advance rapidly. While recent models often achieve higher scores on standard benchmarks, these improvements do not consistently reflect enhanced real-world reasoning capabilities. Moreover, widespread overfitting to public benchmarks and the high computational cost of full evaluations have made it both expensive and less effective to distinguish meaningful differences between models. To address these challenges, we propose the \textbf{S}tructured \textbf{T}ransition \textbf{E}valuation \textbf{M}ethod (STEM), a lightweight and interpretable evaluation framework for efficiently estimating the relative capabilities of LLMs. STEM identifies \textit{significant transition samples} (STS) by analyzing consistent performance transitions among LLMs of the same architecture but varying parameter scales. These samples enable STEM to effectively estimate the capability position of an unknown model. Qwen3 model family is applied to construct the STS pool on six diverse and representative benchmarks. To assess generalizability. Experimental results indicate that STEM reliably captures performance trends, aligns with ground-truth rankings of model capability. These findings highlight STEM as a practical and scalable method for fine-grained, architecture-agnostic evaluation of LLMs.