Do Large Language Models Know What They Are Capable Of?

TL;DR

This paper examines whether large language models can accurately predict their success on tasks, how their self-assessment evolves during multi-step tasks, and how in-context learning influences their decision-making and overconfidence.

Contribution

It provides empirical insights into LLMs' self-assessment abilities, their overconfidence issues, and the impact of in-context experiences on improving their decision-making.

Findings

Most LLMs predict success better than random chance.

Newer and larger LLMs do not necessarily have better self-assessment accuracy.

In-context experiences can reduce overconfidence in some LLMs.

Abstract

We investigate whether large language models (LLMs) can predict whether they will succeed on a given task and whether their predictions improve as they progress through multi-step tasks. We also investigate whether LLMs can learn from in-context experiences to make better decisions about whether to pursue a task in scenarios where failure is costly. All LLMs we tested are overconfident, but most predict their success with better-than-random discriminatory power. We find that newer and larger LLMs generally do not have greater discriminatory power, though Claude models do show such a trend. On multi-step agentic tasks, the overconfidence of several frontier LLMs worsens as they progress through the tasks, and reasoning LLMs perform comparably to or worse than non-reasoning LLMs. With in-context experiences of failure, some but not all LLMs reduce their overconfidence leading to significantly improved decision making, while others do not. Interestingly, all LLMs' decisions are approximately rational given their estimated probabilities of success, yet their overly-optimistic estimates result in poor decision making. These results suggest that current LLM agents are hindered by their lack of awareness of their own capabilities. We discuss the implications of LLMs' awareness of their capabilities for AI misuse and misalignment risks.