Optimizing Chain-of-Thought Confidence via Topological and Dirichlet Risk Analysis

TL;DR

This paper introduces EDTR, a novel method combining topological analysis and Dirichlet uncertainty to improve confidence calibration in large language models during complex reasoning tasks.

Contribution

The paper presents EDTR, a new decoding strategy that uses geometric and topological features to better estimate LLM confidence across multiple reasoning paths.

Findings

EDTR achieves 41% better calibration than existing methods.

EDTR attains perfect accuracy on AIME and high calibration on GSM8K.

EDTR significantly reduces overconfidence in LLM reasoning.

Abstract

Chain-of-thought (CoT) prompting enables Large Language Models to solve complex problems, but deploying these models safely requires reliable confidence estimates, a capability where existing methods suffer from poor calibration and severe overconfidence on incorrect predictions. We propose Enhanced Dirichlet and Topology Risk (EDTR), a novel decoding strategy that combines topological analysis with Dirichlet-based uncertainty quantification to measure LLM confidence across multiple reasoning paths. EDTR treats each CoT as a vector in high-dimensional space and extracts eight topological risk features capturing the geometric structure of reasoning distributions: tighter, more coherent clusters indicate higher confidence while dispersed, inconsistent paths signal uncertainty. We evaluate EDTR against three state-of-the-art calibration methods across four diverse reasoning benchmarks spanning olympiad-level mathematics (AIME), grade school math (GSM8K), commonsense reasoning, and stock price prediction \cite{zhang2025aime, cobbe2021training, talmor-etal-2019-commonsenseqa, yahoo_finance}. EDTR achieves 41\% better calibration than competing methods with an average ECE of 0.287 and the best overall composite score of 0.672, while notably achieving perfect accuracy on AIME and exceptional calibration on GSM8K with an ECE of 0.107, domains where baselines exhibit severe overconfidence. Our work provides a geometric framework for understanding and quantifying uncertainty in multi-step LLM reasoning, enabling more reliable deployment where calibrated confidence estimates are essential.