PaT: Planning-after-Trial for Efficient Test-Time Code Generation

TL;DR

PaT introduces an adaptive, cost-efficient approach for test-time code generation that selectively invokes planning only upon verification failure, significantly reducing inference costs while maintaining high performance.

Contribution

It proposes Planning-after-Trial (PaT), a novel adaptive policy that improves efficiency by combining heterogeneous models and selective planning during test-time code generation.

Findings

Achieves approximately 69% reduction in inference cost.

Significantly improves the cost-performance Pareto frontier.

Maintains performance comparable to larger homogeneous models.

Abstract

Beyond training-time optimization, scaling test-time computation has emerged as a key paradigm to extend the reasoning capabilities of Large Language Models (LLMs). However, most existing methods adopt a rigid Planning-before-Trial (PbT) policy, which inefficiently allocates test-time compute by incurring planning overhead even on directly solvable problems. We propose Planning-after-Trial (PaT), an adaptive policy for code generation that invokes a planner only upon verification failure. This adaptive policy naturally enables a heterogeneous model configuration: a cost-efficient model handles generation attempts, while a powerful model is reserved for targeted planning interventions. Empirically, across multiple benchmarks and model families, our approach significantly advances the cost-performance Pareto frontier. Notably, our heterogeneous configuration achieves performance comparable to a large homogeneous model while reducing inference cost by approximately 69\%.