Scalable Text-Embedding-informed Cognitive Diagnosis of Large Language Models

TL;DR

This paper introduces a scalable, fine-grained evaluation method for large language models using cognitive diagnosis models, incorporating textual information to diagnose strengths and weaknesses across numerous capabilities.

Contribution

It adapts psychometric cognitive diagnosis models for large-scale LLM evaluation, enabling detailed capability profiling with textual priors and efficient estimation algorithms.

Findings

Accurate mastery profile estimation demonstrated in simulations

Effective application to MATH Level 5 benchmark

Uncovered detailed LLM strengths and weaknesses

Abstract

Large language models (LLMs) have achieved remarkable performance on diverse benchmarks, yet existing evaluation practices largely rely on coarse summary metrics that obscure underlying reasoning abilities. In this work, we propose novel methodologies to adapt cognitive diagnosis models (CDMs) in psychometrics to LLM evaluation, enabling fine-grained diagnosis via multidimensional discrete capability profiles and interpretable characterizations of LLM strengths and weaknesses. First, to enable CDM-based evaluation at benchmark scale (more than 1000 items), we propose a scalable method that jointly estimates LLM mastery profiles and the item-attribute Q-matrix, addressing key challenges posed by high-dimensional latent attributes (K > 20), large item pools, and the prohibitive computational cost of existing marginal maximum likelihood-based estimation. Second, we incorporate item-level textual information to construct AI-embedding-informed priors for the Q-matrix, stabilizing high-dimensional estimation while reducing reliance on costly human specification. We develop an efficient stochastic-approximation algorithm to jointly estimate LLM mastery profiles and the Q-matrix that balances data fit with text-embedding-informed priors. Simulation studies demonstrate accurate parameter recovery. An application to the MATH Level 5 benchmark illustrates the practical utility of our method for LLM evaluation and uncovers useful insights into LLMs' fine-grained capabilities.