Multi-Turn Code Generation Through Single-Step Rewards

TL;DR

This paper introduces $ode$, a scalable method for multi-turn code generation that uses only single-step rewards, simplifying the process and improving performance over existing complex hierarchical reinforcement learning approaches.

Contribution

The paper presents $ode$, a novel approach that treats multi-turn code generation as a one-step recoverable MDP, enabling effective training with only single-step rewards.

Findings

$ode$ outperforms state-of-the-art baselines in experiments.

The approach effectively utilizes execution feedback for code generation.

Analysis confirms the simplicity and scalability of the method.

Abstract

We address the problem of code generation from multi-turn execution feedback. Existing methods either generate code without feedback or use complex, hierarchical reinforcement learning to optimize multi-turn rewards. We propose a simple yet scalable approach, $\mu$Code, that solves multi-turn code generation using only single-step rewards. Our key insight is that code generation is a one-step recoverable MDP, where the correct code can be recovered from any intermediate code state in a single turn. $\mu$Code iteratively trains both a generator to provide code solutions conditioned on multi-turn execution feedback and a verifier to score the newly generated code. Experimental evaluations show that our approach achieves significant improvements over the state-of-the-art baselines. We provide analysis of the design choices of the reward models and policy, and show the efficacy of $\mu$Code at utilizing the execution feedback. Our code is available at https://github.com/portal-cornell/muCode.