STAR : Bridging Statistical and Agentic Reasoning for Large Model Performance Prediction

TL;DR

STAR is a novel framework that combines statistical expectations with agentic reasoning to accurately predict large model performance from limited data, improving reliability and interpretability.

Contribution

It introduces a hybrid approach that integrates knowledge retrieval, probabilistic modeling, and reasoning guided by EVT for better performance prediction under data sparsity.

Findings

Achieves 14.46% improvement over statistical methods in sparse settings.

Outperforms all baselines on score-based and rank-based metrics.

Effective with only 1-2 observed scores per model.

Abstract

As comprehensive large model evaluation becomes prohibitively expensive, predicting model performance from limited observations has become essential. However, existing statistical methods struggle with pattern shifts, data sparsity, and lack of explanation, while pure LLM methods remain unreliable. We propose STAR, a framework that bridges data-driven STatistical expectations with knowledge-driven Agentic Reasoning. STAR leverages specialized retrievers to gather external knowledge and embeds semantic features into Constrained Probabilistic Matrix Factorization (CPMF) to generate statistical expectations with uncertainty. A reasoning module guided by Expectation Violation Theory (EVT) then refines predictions through intra-family analysis, cross-model comparison, and credibility-aware aggregation, producing adjustments with traceable explanations. Extensive experiments show that STAR consistently outperforms all baselines on both score-based and rank-based metrics, delivering a 14.46% gain in total score over the strongest statistical method under extreme sparsity, with only 1--2 observed scores per test model.