Steering LLM Thinking with Budget Guidance
Junyan Li, Wenshuo Zhao, Yang Zhang, Chuang Gan
TL;DR
This paper introduces a budget guidance method for large language models that controls reasoning length without fine-tuning, improving token efficiency and accuracy on math benchmarks under tight thinking budgets.
Contribution
It proposes a lightweight predictor-based approach to steer LLM reasoning length, enabling effective budget control without model fine-tuning.
Findings
Achieves up to 26% accuracy improvement under tight budgets.
Reduces reasoning tokens to 63% of full-thinking models.
Generalizes to diverse tasks and estimates question difficulty.
Abstract
Recent deep-thinking large language models often reason extensively to improve performance, but such lengthy reasoning is not always desirable, as it incurs excessive inference costs with disproportionate performance gains. Controlling reasoning length without sacrificing performance is therefore important, but remains challenging, especially under tight thinking budgets. We propose budget guidance, a simple yet effective method for steering the reasoning process of LLMs toward a target budget without requiring any LLM fine-tuning. Our approach introduces a lightweight predictor that models a Gamma distribution over the remaining thinking length during next-token generation. This signal is then used to guide generation in a soft, token-level manner, ensuring that the overall reasoning trace adheres to the specified thinking budget. Budget guidance enables natural control of the thinking…
Peer Reviews
Decision·ICLR 2026 Conference Withdrawn Submission
This paper proposes a budget-aware controllable LLM next-token-prediction for efficient thinking. The authors propose to directly model the remaining thinking length as a Gamma distribution and use the CDF of Gamma distribution to compute the predictor score, which are used to steer the final LLM token distribution. The normalized product of $u_t$ and $a_t$ is defined as the conditional vector.
The underlying motivations of using the Gamma distribution to model the remaining length and using the CDF of it as the generated steering vectors to guide the final output decoding of LLMs are not clearly explained. The experiments are not thoroughly conducted with more missing SOTA training-free baselines. In addition, the accuracy degradation of three datasets are not ignorable. The end-2-end runtime reduction and the additional cost of embedding and BERT inference are not reported.
1）The proposed “budget guidance” is a simple yet effective inference-time method that enables flexible control over chain-of-thought (CoT) length without supervised fine-tuning (SFT), avoiding computational costs and potential safety risks associated with LLM fine-tuning. 2）The lightweight predictor is innovative, as it models remaining reasoning length distributionally and generalizes well to out-of-domain tasks (e.g., scientific and logical reasoning), highlighting its strong adaptability. 3
The study lacks a prompt engineering (PE) baseline. Given the insights that the predictor is prompt-aware and capable of adapting to instructions (e.g., generating concise or lengthy reasoning), a comparison with well-crafted prompts could validate whether similar length-control effects can be achieved without the predictor, strengthening the claim of necessity.
- The proposed budget-conditional output distribution is interesting, and the framework has a principled design. - The trained predictor shows task generalization, even though it is only trained on math. - The paper is easy to understand.
- The evaluated baselines only include NoThinking and Budget Forcing, which are not representative and strong enough. There are many other existing adaptive thinking methods, but the authors did not review or compare their method against other adaptive thinking methods. This makes the empirical evaluation of the proposed method weak. I would suggest the other at least compare with two other adaptive thinking methods to show the real effective of the proposed method. - If taking the last token em
1. The paper introduces a fine-tuning–free approach to guide the reasoning process of large language models (LLMs), which can be seamlessly integrated as a plugin into modern LLM serving systems. 2. The proposed Budget Guidance mechanism applies a soft, token-level steering that effectively regulates reasoning length while preserving the model’s inherent reasoning capability. 3. The predictor operates only once at the start of each reasoning segment, resulting in negligible latency overhead and
1. Limited baselines: The paper only compares against one budget-based method. Missing broader hybrid-thinking or trade-off baselines, such as prompt-based classification or adaptive fast/slow reasoning strategies. 2. Lack of diversity in model structure: All tested models (DeepSeek-R1-7B/32B, Qwen3-8B) are dense models. It’s unclear whether Budget Guidance remains effective on MoE architectures. 3. Single token budget objective setting: Evaluations only target at half the original model’s full
This paper proposes a budget guidance method for steering the reasoning process of LLMs without requiring any LLM fine-tuning. It can achieve up to a 26% accuracy gain on the MATH-500 benchmark under tight budgets compared to baseline methods, while maintaining competitive accuracy with only 63% of the thinking tokens used by the full-thinking model. Different benchmarks have been tested.
1. The method's core relies on modeling the remaining thinking length with a Gamma distribution. This is a strong assumption about the nature of the reasoning process. 2. The paper compares against "baseline methods," but it's important to define these and include more sophisticated alternatives. So many papers have discussed on overthinking and underthinking. More experiments should be added.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsERP Systems Implementation and Impact