Can Confidence Estimates Decide When Chain-of-Thought Is Necessary for LLMs?

TL;DR

This paper investigates confidence-based gating to determine when large language models should use chain-of-thought reasoning, aiming to reduce unnecessary computation while maintaining reasoning accuracy.

Contribution

It introduces a systematic evaluation framework for confidence signals and compares multiple measures for selective reasoning in LLMs, providing practical guidance.

Findings

Confidence measures can reduce redundant reasoning

Performance of confidence signals varies across tasks

Existing measures are training-free and effective in some settings

Abstract

Chain-of-thought (CoT) prompting is a common technique for improving the reasoning abilities of large language models (LLMs). However, extended reasoning is often unnecessary and substantially increases token usage. As such, a key question becomes how to optimally allocate compute to when reasoning is actually needed. We study this through confidence-gated CoT, where a model produces a direct answer and a confidence estimate to decide whether to invoke CoT. We present an evaluation framework together with the first systematic study of confidence signals for this decision. We evaluate four representative confidence measures and compare them with random gating and an oracle upper bound. Experiments across two model families and diverse reasoning tasks show that existing training-free confidence measures can reduce redundant reasoning. However, we also find that the utility of individual confidence measures is inconsistent across settings. Through our evaluation framework and analysis, our study provides practical guidance toward developing and evaluating models that selectively use CoT.