Self-Consistency from Only Two Samples: CoT-PoT Ensembling for Efficient LLM Reasoning

TL;DR

This paper introduces CoT-PoT ensembling, a hybrid approach combining Chain-of-Thought and Program-of-Thought reasoning, which significantly reduces sampling costs and enables high accuracy with only two samples.

Contribution

The authors propose a novel hybrid ensembling framework that integrates CoT and PoT reasoning modes, achieving high accuracy with minimal samples in self-consistency methods.

Findings

Reduces the number of samples needed for self-consistency by 9.3 times.

Most tasks (78.6%) can be solved with only two samples.

Improves reasoning accuracy while lowering computational costs.

Abstract

Self-consistency (SC) is a popular technique for improving the reasoning accuracy of large language models by aggregating multiple sampled outputs, but it comes at a high computational cost due to extensive sampling. We introduce a hybrid ensembling approach that leverages the complementary strengths of two distinct modes of reasoning: Chain-of-Thought (CoT) and Program-of-Thought (PoT). We describe a general framework for combining these two forms of reasoning in self-consistency, as well as particular strategies for both full sampling and early-stopping. We show that CoT-PoT ensembling not only improves overall accuracy, but also drastically reduces the number of samples required for SC by a factor of 9.3x. In particular, the majority of tasks (78.6%) can be addressed with only two samples, which has not been possible with any prior SC methods.