The Dunning-Kruger Effect in Large Language Models: An Empirical Study of Confidence Calibration

TL;DR

This study empirically investigates whether large language models exhibit the Dunning-Kruger effect, revealing that less accurate models tend to be more overconfident, which has important implications for their safe deployment.

Contribution

The paper provides the first empirical evidence of Dunning-Kruger-like bias in LLMs, analyzing calibration patterns across multiple models and datasets.

Findings

Poorly performing models show higher overconfidence.

Calibration accuracy varies significantly among models.

Overconfidence correlates with lower task accuracy.

Abstract

Large language models (LLMs) have demonstrated remarkable capabilities across diverse tasks, yet their ability to accurately assess their own confidence remains poorly understood. We present an empirical study investigating whether LLMs exhibit patterns reminiscent of the Dunning-Kruger effect -- a cognitive bias where individuals with limited competence tend to overestimate their abilities. We evaluate four state-of-the-art models (Claude Haiku 4.5, Gemini 2.5 Pro, Gemini 2.5 Flash, and Kimi K2) across four benchmark datasets totaling 24,000 experimental trials. Our results reveal striking calibration differences: Kimi K2 exhibits severe overconfidence with an Expected Calibration Error (ECE) of 0.726 despite only 23.3% accuracy, while Claude Haiku 4.5 achieves the best calibration (ECE = 0.122) with 75.4% accuracy. These findings demonstrate that poorly performing models display markedly higher overconfidence -- a pattern analogous to the Dunning-Kruger effect in human cognition. We discuss implications for safe deployment of LLMs in high-stakes applications.