Reasoning Through Execution: Unifying Process and Outcome Rewards for Code Generation

TL;DR

This paper introduces Outcome Refining Process Supervision (ORPS), a novel framework that combines process and outcome supervision using executable verification to improve large language models' ability to generate correct and efficient code for complex tasks.

Contribution

The paper proposes ORPS, a unified supervision method that leverages executable verification, runtime feedback, and self-critique to enhance code generation in large language models.

Findings

26.9% higher correctness in code generation

42.2% improved code efficiency

Consistent performance gains across models and benchmarks

Abstract

Large Language Models excel at code generation yet struggle with complex programming tasks that demand sophisticated reasoning. To bridge this gap, traditional process supervision relies on learned reward models requiring costly training data and suffering from reward misalignment, while outcome supervision fails for complex tasks needing coordinated intermediate steps. We introduce Outcome Refining Process Supervision, which unifies process and outcome supervision by leveraging executable verification: a tree-structured search framework generates strategic alternatives, profiles execution metrics, and scores candidates via self-critique mechanisms that integrate runtime feedback with reasoning. Experiments across 5 models and 3 benchmarks show consistent gains, with 26.9% higher correctness and 42.2% improved code efficiency. The results demonstrate that ORPS enables LLMs to overcome local optima in code generation, suggesting a promising direction for combining verifiable outcomes with structured reasoning to tackle complex challenges. We open-source at: https://github.com/zhuohaoyu/ORPS