Make Every Penny Count: Difficulty-Adaptive Self-Consistency for Cost-Efficient Reasoning

TL;DR

This paper introduces Difficulty-Adaptive Self-Consistency (DSC), a novel method that uses question difficulty information to dynamically allocate reasoning resources, significantly reducing costs while maintaining performance in multi-step reasoning tasks.

Contribution

The paper proposes DSC, which leverages prior and posterior difficulty information to adaptively allocate inference resources, outperforming existing methods in cost-efficiency.

Findings

DSC reduces reasoning costs significantly compared to ASC and ESC.

DSC maintains comparable accuracy across arithmetic, commonsense, and symbolic reasoning tasks.

Extensive experiments validate the effectiveness of DSC on six benchmarks.

Abstract

Self-consistency (SC), a widely used decoding strategy for chain-of-thought reasoning, shows significant gains across various multi-step reasoning tasks but comes with a high cost due to multiple sampling with the preset size. Its variants, Adaptive self-consistency (ASC) and Early-stopping self-consistency (ESC), dynamically adjust the number of samples based on the posterior distribution of a set of pre-samples, reducing the cost of SC with minimal impact on performance. Both methods, however, do not exploit the prior information about question difficulty. It often results in unnecessary repeated sampling for easy questions that could be accurately answered with just one attempt, wasting resources. To tackle this problem, we propose Difficulty-Adaptive Self-Consistency (DSC), which leverages the difficulty information of batch queries from both prior and posterior perspectives to adaptively allocate inference resources, further reducing the overall cost of SC. To demonstrate the effectiveness of DSC, we conduct extensive experiments on three popular categories of reasoning tasks: arithmetic, commonsense and symbolic reasoning on six benchmarks. The empirical results show that DSC consistently surpasses the strong baseline ASC and ESC in terms of costs by a significant margin, while attaining comparable performances.