Long or short CoT? Investigating Instance-level Switch of Large Reasoning Models

TL;DR

This paper analyzes the trade-offs between long and short Chain-of-Thought prompting in large reasoning models, proposing an adaptive framework that balances accuracy and efficiency based on resource constraints.

Contribution

It introduces SwitchCoT, an adaptive, budget-aware framework that dynamically selects between long and short CoT strategies to optimize reasoning performance and computational cost.

Findings

Long CoT outperforms with ample resources.

Short CoT is more efficient under tight budgets.

SwitchCoT reduces inference costs by up to 50%.

Abstract

With the rapid advancement of large reasoning models, long Chain-of-Thought (CoT) prompting has demonstrated strong performance on complex tasks. However, this often comes with a significant increase in token usage. In this paper, we conduct a comprehensive empirical analysis comparing long and short CoT strategies. Our findings reveal that while long CoT can lead to performance improvements, its benefits are often marginal relative to its significantly higher token consumption. Specifically, long CoT tends to outperform when ample generation budgets are available, whereas short CoT is more effective under tighter budget constraints. These insights underscore the need for a dynamic approach that selects the proper CoT strategy based on task context and resource availability. To address this, we propose SwitchCoT, an automatic framework that adaptively chooses between long and short CoT strategies to balance reasoning accuracy and computational efficiency. Moreover, SwitchCoT is designed to be budget-aware, making it broadly applicable across scenarios with varying resource constraints. Experimental results demonstrate that SwitchCoT can reduce inference costs by up to 50% while maintaining high accuracy. Notably, under limited token budgets, it achieves performance comparable to, or even exceeding, that of using either long or short CoT alone.