LLMs as Signal Detectors: Sensitivity, Bias, and the Temperature-Criterion Analogy

TL;DR

This study applies Signal Detection Theory to analyze large language models as signal detectors, revealing nuanced insights into their sensitivity and bias that are not captured by traditional calibration metrics.

Contribution

It introduces the use of full SDT parametric analysis to LLMs, demonstrating the breakdown of the temperature-criterion analogy and providing a more detailed diagnostic framework.

Findings

Temperature increases sensitivity and shifts criterion simultaneously.

Models show unequal-variance distributions with varying asymmetry.

SDT decomposition distinguishes models beyond calibration metrics.

Abstract

Large language models (LLMs) are evaluated for calibration using metrics such as Expected Calibration Error that conflate two distinct components: the model's ability to discriminate correct from incorrect answers (sensitivity) and its tendency toward confident or cautious responding (bias). Signal Detection Theory (SDT) decomposes these components. While SDT-derived metrics such as AUROC are increasingly used, the full parametric framework - unequal-variance model fitting, criterion estimation, z-ROC analysis - has not been applied to LLMs as signal detectors. In this pre-registered study, we treat three LLMs as observers performing factual discrimination across 168,000 trials and test whether temperature functions as a criterion shift analogous to payoff manipulations in human psychophysics. Critically, this analogy may break down because temperature changes the generated answer itself, not only the confidence assigned to it. Our results confirm the breakdown with temperature simultaneously increasing sensitivity (AUC) and shifting criterion. All models exhibited unequal-variance evidence distributions (z-ROC slopes 0.52-0.84), with instruct models showing more extreme asymmetry (0.52-0.63) than the base model (0.77-0.87) or human recognition memory (~0.80). The SDT decomposition revealed that models occupying distinct positions in sensitivity-bias space could not be distinguished by calibration metrics alone, demonstrating that the full parametric framework provides diagnostic information unavailable from existing metrics.