Efficient Thought Space Exploration Through Strategic Intervention

TL;DR

This paper introduces HPR, a framework that improves reasoning efficiency in language models by strategically intervening at critical decision points, reducing computational costs while maintaining high accuracy.

Contribution

The paper presents a novel Hint-Practice Reasoning framework with a distributional inconsistency reduction metric for dynamic intervention, enhancing reasoning efficiency and accuracy.

Findings

HPR achieves comparable accuracy with 1/5 tokens decoded.

HPR outperforms existing methods by up to 5.1% in accuracy.

HPR maintains similar or lower FLOPs compared to baselines.

Abstract

While large language models (LLMs) demonstrate emerging reasoning capabilities, current inference-time expansion methods incur prohibitive computational costs by exhaustive sampling. Through analyzing decoding trajectories, we observe that most next-token predictions align well with the golden output, except for a few critical tokens that lead to deviations. Inspired by this phenomenon, we propose a novel Hint-Practice Reasoning (HPR) framework that operationalizes this insight through two synergistic components: 1) a hinter (powerful LLM) that provides probabilistic guidance at critical decision points, and 2) a practitioner (efficient smaller model) that executes major reasoning steps. The framework's core innovation lies in Distributional Inconsistency Reduction (DIR), a theoretically-grounded metric that dynamically identifies intervention points by quantifying the divergence between practitioner's reasoning trajectory and hinter's expected distribution in a tree-structured probabilistic space. Through iterative tree updates guided by DIR, HPR reweights promising reasoning paths while deprioritizing low-probability branches. Experiments across arithmetic and commonsense reasoning benchmarks demonstrate HPR's state-of-the-art efficiency-accuracy tradeoffs: it achieves comparable performance to self-consistency and MCTS baselines while decoding only 1/5 tokens, and outperforms existing methods by at most 5.1% absolute accuracy while maintaining similar or lower FLOPs.