The Virtues of Brevity: Avoid Overthinking in Parallel Test-Time Reasoning

TL;DR

This paper shows that choosing the shortest solution in parallel test-time reasoning with LLMs is a simple yet effective heuristic that improves performance and reduces computational costs by favoring confident, concise answers over verbose overthinking.

Contribution

Demonstrates that a simple shortest-answer heuristic can match complex methods like self-consistency, offering a computationally efficient alternative for improving LLM reasoning performance.

Findings

Shortest-answer heuristic is highly effective in parallel reasoning.

The approach is competitive with complex methods on challenging benchmarks.

It provides a Pareto improvement by balancing performance and computational cost.

Abstract

Reasoning models represent a significant advance in LLM capabilities, particularly for complex reasoning tasks such as mathematics and coding. Previous studies confirm that parallel test-time compute-sampling multiple solutions and selecting the best one-can further enhance the predictive performance of LLMs. However, strategies in this area often require complex scoring, thus increasing computational cost and complexity. In this work, we demonstrate that the simple and counterintuitive heuristic of selecting the shortest solution is highly effective. We posit that the observed effectiveness stems from models operating in two distinct regimes: a concise, confident conventional regime and a verbose overthinking regime characterized by uncertainty, and we show evidence of a critical point where the overthinking regime begins to be significant. By selecting the shortest answer, the heuristic preferentially samples from the conventional regime. We confirm that this approach is competitive with more complex methods such as self-consistency across two challenging benchmarks while significantly reducing computational overhead. The shortest-answer heuristic provides a Pareto improvement over self-consistency and applies even to tasks where output equality is not well defined.