Pause and Reflect: Conformal Aggregation for Chain-of-Thought Reasoning
Yu Gu, Zijun Yu, Vahid Partovi Nia, Masoud Asgharian

TL;DR
This paper introduces a conformal aggregation method for chain-of-thought reasoning that provides finite-sample guarantees on confident-error rates and improves selective accuracy without retraining.
Contribution
It proposes a conformal procedure replacing majority voting with weighted score aggregation and calibrates abstention rules, ensuring reliable confidence guarantees in reasoning tasks.
Findings
Achieves 90.1% selective accuracy on GSM8K by abstaining on less than 5% of problems.
Provides finite-sample guarantees on confident-error rates.
Outperforms majority-voting baseline with 82% accuracy on GSM8K.
Abstract
Chain-of-thought (CoT) reasoning with self-consistency improves performance by aggregating multiple sampled reasoning paths. In this setting, correctness is no longer tied to a single reasoning trace but to the aggregation rule over a pool of candidate paths, making aggregation uncertainty the central challenge. This issue is critical where confidently incorrect answers are far more costly than abstentions. We introduce a conformal procedure for CoT reasoning that directly addresses aggregation uncertainty. Our approach replaces majority voting with weighted score aggregation over reasoning paths and calibrates an abstention rule using conformal risk control. This approach leads to finite-sample guarantees on the confident-error rate--the probability that the system answers and is wrong. We further identify score separability as the key condition under which abstention provably improves…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
