CreditAudit: 2$^\text{nd}$ Dimension for LLM Evaluation and Selection

TL;DR

CreditAudit introduces a new evaluation framework for language models that assesses both average performance and stability across different prompts, aiding more reliable deployment decisions.

Contribution

It proposes a deployment-oriented, 2D evaluation method that considers model stability and performance, improving upon traditional single-score benchmarks.

Findings

Models with similar average scores can have different stability profiles.

Stability risk can change model prioritization in high-stakes scenarios.

Credit grades help interpret model reliability and guide deployment choices.

Abstract

Leaderboard scores on public benchmarks have been steadily rising and converging, with many frontier language models now separated by only marginal differences. However, these scores often fail to match users' day to day experience, because system prompts, output protocols, and interaction modes evolve under routine iteration, and in agentic multi step pipelines small protocol shifts can trigger disproportionate failures, leaving practitioners uncertain about which model to deploy. We propose CreditAudit, a deployment oriented credit audit framework that evaluates models under a family of semantically aligned and non adversarial system prompt templates across multiple benchmarks, reporting mean ability as average performance across scenarios and scenario induced fluctuation sigma as a stability risk signal, and further mapping volatility into interpretable credit grades from AAA to BBB via cross model quantiles with diagnostics that mitigate template difficulty drift. Controlled experiments on GPQA, TruthfulQA, and MMLU Pro show that models with similar mean ability can exhibit substantially different fluctuation, and stability risk can overturn prioritization decisions in agentic or high failure cost regimes. By providing a 2D and grade based language for regime specific selection, CreditAudit supports tiered deployment and more disciplined allocation of testing and monitoring effort, enabling more objective and trustworthy model evaluation for real world use.