Beyond Execution: Static-Analysis Rewards and Hint-Conditioned Diffusion RL for Code Generation

TL;DR

This paper systematically studies reinforcement learning strategies for diffusion-based code generation, emphasizing reward design, hint conditioning, and task difficulty, with static checking emerging as a strong execution-free reward.

Contribution

It provides empirical insights into how reward design and hint conditioning influence diffusion RL performance across varying task difficulties.

Findings

Static checking is the most effective execution-free reward, significantly improving code generation metrics.

Moderate AST-based hinting benefits harder benchmarks, aiding exploration.

Reward effectiveness varies with task difficulty, with static checking excelling on challenging tasks.

Abstract

Reinforcement Learning (RL) is an important paradigm for aligning Diffusion Language Models (DLMs) toward functional correctness in code generation. However, these models often encounter a ``capability cliff'' on complex tasks, where execution-based semantic rewards become too low to provide a viable learning signal. In this paper, we present a systematic empirical study of RL post-training for diffusion-based code generation along three axes: reward design, hint-conditioned sampling, and task difficulty. We investigate the effectiveness of execution-free rewards as alternatives to traditional unit-test execution, the role of training-time hint-conditioned diffusion sampling in mitigating exploration bottlenecks, and the impact of these design choices varies across tasks with different difficulty levels. Across HumanEval, MBPP, and LiveCodeBench, we find that static checking is the strongest overall standalone execution-free reward in our setting, especially improving DiffuCoder from 53.9 to 67.1 on HumanEval and from 14.9 to 15.5 on LiveCodeBench while reducing rollout time by 9.4\%. We further find that moderate AST-based hinting is most useful on harder benchmarks, while the best reward design depends strongly on task difficulty: similarity-based rewards are more effective on easier subsets, whereas static checking is more reliable on harder subsets where execution rewards are low. These findings suggest that reward design and training guidance substantially affect diffusion RL performance in our evaluated code-generation setting.