Policy of Thoughts: Scaling LLM Reasoning via Test-time Policy Evolution

TL;DR

This paper introduces Policy of Thoughts (PoT), a novel framework that dynamically refines large language models' reasoning strategies during test time by learning from execution feedback, significantly improving complex reasoning performance.

Contribution

PoT recasts reasoning as an online optimization process, enabling real-time policy evolution within instances, which is a novel approach compared to prior static or external feedback methods.

Findings

PoT achieves 49.71% accuracy on LiveCodeBench with a 4B model.

PoT outperforms GPT-4o and DeepSeek-V3 despite smaller size.

Dynamic policy refinement improves reasoning accuracy significantly.

Abstract

Large language models (LLMs) struggle with complex, long-horizon reasoning due to instability caused by their frozen policy assumption. Current test-time scaling methods treat execution feedback merely as an external signal for filtering or rewriting trajectories, without internalizing it to improve the underlying reasoning strategy. Inspired by Popper's epistemology of "conjectures and refutations," we argue that intelligence requires real-time evolution of the model's policy through learning from failed attempts. We introduce Policy of Thoughts (PoT), a framework that recasts reasoning as a within-instance online optimization process. PoT first generates diverse candidate solutions via an efficient exploration mechanism, then uses Group Relative Policy Optimization (GRPO) to update a transient LoRA adapter based on execution feedback. This closed-loop design enables dynamic, instance-specific refinement of the model's reasoning priors. Experiments show that PoT dramatically boosts performance: a 4B model achieves 49.71% accuracy on LiveCodeBench, outperforming GPT-4o and DeepSeek-V3 despite being over 50 smaller.