Think or Not? Exploring Thinking Efficiency in Large Reasoning Models via an Information-Theoretic Lens

TL;DR

This paper analyzes the efficiency of reasoning in large models using information theory, revealing a trade-off between reasoning length and semantic quality, and proposes an entropy-based adaptive strategy to improve efficiency and accuracy.

Contribution

It introduces InfoBias and InfoGain metrics to quantify reasoning quality and proposes an entropy-based adaptive halt strategy for large reasoning models.

Findings

Longer reasoning chains have higher information bias and diminishing information gain.

The adaptive strategy improves accuracy by 1.10% and reduces token usage by 50.80%.

The method enhances reasoning efficiency across diverse tasks.

Abstract

The recent rise of Large Reasoning Models (LRMs) has significantly improved multi-step reasoning performance, but often at the cost of generating excessively long reasoning chains. This paper revisits the efficiency of such reasoning processes through an information-theoretic lens, revealing a fundamental trade-off between reasoning length and semantic efficiency. We propose two metrics, InfoBias and InfoGain, to quantify divergence from ideal reasoning paths and stepwise information contribution, respectively. Empirical analyses show that longer reasoning chains tend to exhibit higher information bias and diminishing information gain, especially for incorrect answers. Motivated by these findings, we introduce an entropy-based Adaptive Think strategy that dynamically halts reasoning once confidence is sufficiently high, improving efficiency while maintaining competitive accuracy. Compared to the Vanilla Think approach (default mode), our strategy yields a 1.10% improvement in average accuracy and a 50.80% reduction in token usage on QwQ-32B across six benchmark tasks spanning diverse reasoning types and difficulty levels, demonstrating superior efficiency and reasoning performance. These results underscore the promise of entropy-based methods for enhancing both accuracy and cost-effiiciency in large language model deployment.