Diagnosing Multi-step Reasoning Failures in Black-box LLMs via Stepwise Confidence Attribution

TL;DR

This paper introduces Stepwise Confidence Attribution (SCA), a novel framework for diagnosing reasoning failures in black-box large language models by assigning confidence to individual reasoning steps without internal model access.

Contribution

The paper proposes SCA, a new method applying the Information Bottleneck principle to assign step-level confidence in black-box LLMs, enabling better diagnosis and correction of reasoning errors.

Findings

SCA reliably identifies low-confidence, error-prone reasoning steps.

Using step-level confidence improves self-correction success rate by up to 13.5%.

Experiments on mathematical reasoning and multi-hop QA validate effectiveness.

Abstract

Large Language Models have achieved strong performance on reasoning tasks with objective answers by generating step-by-step solutions, but diagnosing where a multi-step reasoning trace might fail remains difficult. Confidence estimation offers a diagnostic signal, yet existing methods are restricted to final answers or require internal model access. In this paper, we introduce Stepwise Confidence Attribution (SCA), a framework for closed-source LLMs that assigns step-level confidence based only on generated reasoning traces. SCA applies the Information Bottleneck principle: steps aligning with consensus structures across correct solutions receive high confidence, while deviations are flagged as potentially erroneous. We propose two complementary methods: (1) NIBS, a non-parametric IB approach measuring consistency without graph structures, and (2) GIBS, a graph-based IB model that learns subgraphs through a differentiable mask to capture logical variability. Extensive experiments on mathematical reasoning and multi-hop question answering show that SCA reliably identifies low-confidence steps strongly correlated with reasoning errors. Moreover, using step-level confidence to guide self-correction improves the correction success rate by up to 13.5\% over answer-level feedback.