Don't Think Longer, Think Wisely: Optimizing Thinking Dynamics for Large Reasoning Models

TL;DR

This paper introduces a dynamic reasoning path optimization framework for large reasoning models that reduces computational costs and improves accuracy by promoting beneficial thinking patterns and eliminating unnecessary overthinking.

Contribution

It proposes a novel method to segment and optimize reasoning paths in LRMs, enhancing efficiency and accuracy through dynamic pattern selection and preference optimization.

Findings

Reduces attention FLOPs by up to 47% while maintaining accuracy.

Transforms some incorrect responses into correct ones, improving accuracy by 15.6%.

Decreases token usage from 5,000 to 3,000 tokens, with up to 12% accuracy gain.

Abstract

While recent success of large reasoning models (LRMs) significantly advanced LLMs' reasoning capability by optimizing the final answer accuracy using reinforcement learning, they may also drastically increase the output length due to overthinking, characterized by unnecessarily complex reasoning paths that waste computation and potentially degrade the performance. We hypothesize that such inefficiencies stem from LRMs' limited capability to dynamically select the proper modular reasoning strategies, termed thinking patterns at the right position. To investigate this hypothesis, we propose a dynamic optimization framework that segments model-generated reasoning paths into distinct thinking patterns, systematically identifying and promoting beneficial patterns that improve the answer while removing detrimental ones. Empirical analysis confirms that our optimized thinking paths yield more concise yet sufficiently informative trajectories, enhancing reasoning efficiency by reducing attention FLOPs by up to 47% while maintaining accuracy for originally correct responses. Moreover, a non-trivial portion of originally incorrect responses are transformed into correct ones, achieving a 15.6% accuracy improvement with reduced length. Motivated by the improvement brought by the optimized thinking paths, we apply a preference optimization technique supported by a pairwise dataset contrasting suboptimal and optimal reasoning paths. Experimental evaluations across multiple mathematical reasoning benchmarks reveal that our method notably reduces computational overhead while simultaneously improving reasoning accuracy, achieving up to a 12% accuracy improvement and reducing token usage from approximately 5,000 to 3,000 tokens.