Beyond Confidence: Rethinking Self-Assessments for Performance Prediction in LLMs

TL;DR

This paper introduces a multidimensional self-assessment framework for LLMs, demonstrating that appraisal-based measures like effort and ability outperform confidence in predicting model failure across diverse tasks.

Contribution

It proposes a novel multidimensional self-assessment approach based on cognitive appraisal theory, enhancing reliability of performance prediction in LLMs.

Findings

Competence-related appraisals like effort and ability outperform confidence in failure prediction.

Effort provides stable, less overoptimistic estimates across model sizes.

Task characteristics influence which appraisal dimension is most predictive.

Abstract

Large Language Models (LLMs) are increasingly used in settings where reliable self-assessment is critical. Assessing model reliability has evolved from using probabilistic correctness estimates to, more recently, eliciting verbalized confidence. Confidence, however, has been shown to be an inconsistent and overoptimistic predictor of model correctness. Drawing on cognitive appraisal theory, a framework from human psychology that decomposes self-evaluation into multiple components, we propose a multidimensional perspective on model self-assessment. We elicit six appraisal-based dimensions of self-assessment, alongside confidence, and evaluate their utility for predicting model failure across 12 LLMs and 38 tasks spanning eight domains. We find that competence-related appraisal dimensions, particularly effort and ability, consistently match or outperform confidence across most settings. Effort additionally yields less overoptimistic estimates that remain stable across model sizes. In contrast, affective dimensions provide marginally predictive signals. Furthermore, the most informative dimension varies systematically with task characteristics: effort is most predictive for reasoning-intensive tasks, while ability and confidence dominate on retrieval-oriented tasks. Broadly, our findings indicate that structured multidimensional self-assessment is a promising approach to improving the reliability and safety of language model deployment across diverse real-world settings.