Thinking Fast and Right: Balancing Accuracy and Reasoning Length with Adaptive Rewards

TL;DR

This paper introduces an adaptive reward-shaping technique for large language models that balances reasoning accuracy and length, reducing inference costs by producing concise, correct responses through dynamic reward adjustments.

Contribution

It presents a novel adaptive reward method that dynamically balances accuracy and length, improving reasoning efficiency without sacrificing correctness.

Findings

Significantly reduces reasoning length across multiple datasets.

Maintains high accuracy despite shorter reasoning traces.

Accelerates early-stage length reduction without over-compression.

Abstract

Large language models (LLMs) have demonstrated strong reasoning abilities in mathematical tasks, often enhanced through reinforcement learning (RL). However, RL-trained models frequently produce unnecessarily long reasoning traces -- even for simple queries -- leading to increased inference costs and latency. While recent approaches attempt to control verbosity by adding length penalties to the reward function, these methods rely on fixed penalty terms that are hard to tune and cannot adapt as the model's reasoning capability evolves, limiting their effectiveness. In this work, we propose an adaptive reward-shaping method that enables LLMs to "think fast and right" -- producing concise outputs without sacrificing correctness. Our method dynamically adjusts the reward trade-off between accuracy and response length based on model performance: when accuracy is high, the length penalty increases to encourage faster length reduction; when accuracy drops, the penalty is relaxed to preserve correctness. This adaptive reward accelerates early-stage length reduction while avoiding over-compression in later stages. Experiments across multiple datasets show that our approach consistently and dramatically reduces reasoning length while largely maintaining accuracy, offering a new direction for cost-efficient adaptive reasoning in large-scale language models.