Latency-Response Theory Model: Evaluating Large Language Models via Response Accuracy and Chain-of-Thought Length

TL;DR

This paper introduces Latency-Response Theory (LaRT), a novel evaluation framework for large language models that jointly models response accuracy and chain-of-thought length, providing more precise assessments than existing methods.

Contribution

LaRT extends Item Response Theory by incorporating response latency, modeling the correlation between reasoning ability and speed, and demonstrating improved estimation accuracy and evaluation metrics.

Findings

LaRT outperforms IRT in estimation accuracy and confidence interval precision.

A strong negative correlation between ability and speed is observed across benchmarks.

LaRT provides different and more reliable LLM rankings than IRT.

Abstract

The proliferation of Large Language Models (LLMs) necessitates valid evaluation methods to guide downstream applications and actionable future improvements. The Item Response Theory (IRT) has recently emerged as a promising framework for evaluating LLMs via their response accuracy. Beyond simple response accuracy, LLMs' chain of thought (CoT) lengths serve as a vital indicator of their reasoning ability. To leverage the CoT length information to assist the evaluation of LLMs, we propose Latency-Response Theory (LaRT) to jointly model the response accuracy and CoT length by introducing the latent ability, latent speed, and a key correlation parameter between them. We derive an efficient estimation algorithm and establish rigorous identifiability results for the population parameters to ensure the statistical validity of estimation. Theoretical asymptotic analyses and simulation studies demonstrate LaRT's advantages over IRT in terms of higher estimation accuracy and shorter confidence intervals for latent traits. A key finding is that the asymptotic estimation precision of the latent ability under LaRT exceeds that of IRT whenever the latent ability and latent speed are correlated. We collect real responses from diverse LLMs on popular benchmark datasets. The application of LaRT reveals a strong negative correlation between the latent ability and latent speed in all benchmarks, with stronger correlation for more difficult benchmarks. This finding supports the intuition that higher reasoning ability correlates with slower speed and longer response latency. LaRT yields different LLM rankings than IRT and outperforms IRT across multiple key evaluation metrics including predictive power, item efficiency, ranking validity, and LLM evaluation efficiency. Code and data are available at https://github.com/Toby-X/Latency-Response-Theory-Model.