Escape Sky-high Cost: Early-stopping Self-Consistency for Multi-step Reasoning

TL;DR

This paper introduces ESC, an early-stopping self-consistency method that significantly reduces the sampling cost in multi-step reasoning tasks without sacrificing accuracy, demonstrated across various benchmarks.

Contribution

ESC provides a simple, scalable approach to reduce the cost of self-consistency in reasoning, with a dynamic control scheme for balancing performance and efficiency.

Findings

ESC reduces sampling by up to 84% across benchmarks.

ESC maintains comparable performance with fewer samples.

Extensive experiments validate ESC's effectiveness on multiple reasoning tasks.

Abstract

Self-consistency (SC) has been a widely used decoding strategy for chain-of-thought reasoning. Despite bringing significant performance improvements across a variety of multi-step reasoning tasks, it is a high-cost method that requires multiple sampling with the preset size. In this paper, we propose a simple and scalable sampling process, \textbf{E}arly-Stopping \textbf{S}elf-\textbf{C}onsistency (ESC), to greatly reduce the cost of SC without sacrificing performance. On this basis, one control scheme for ESC is further derivated to dynamically choose the performance-cost balance for different tasks and models. To demonstrate ESC's effectiveness, we conducted extensive experiments on three popular categories of reasoning tasks: arithmetic, commonsense and symbolic reasoning over language models with varying scales. The empirical results show that ESC reduces the average number of sampling of chain-of-thought reasoning by a significant margin on six benchmarks, including MATH (-33.8%), GSM8K (-80.1%), StrategyQA (-76.8%), CommonsenseQA (-78.5%), Coin Flip (-84.2%) and Last Letters (-67.4%), while attaining comparable performances.